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Overview 


This  report  is  the  annual  report  for  Phase  2  of  the  Interactive  Model-Centric  Systems  Engineering 
(IMCSE)  research  project.  Portions  of  the  Phase  1  report  have  been  repeated  in  this  report  for 
completeness. 


Purpose  of  Research 

Model  based  systems  engineering  (MBSE)  is  becoming  increasingly  more  important  in  the 
practice  of  SE.  MBSE  methods  and  tools  are  used  throughout  the  entire  lifecycle  to  generate 
systems,  software  and  hardware  products,  and  work  towards  replacing  labor-intensive  and  error- 
prone  documentation-based  processes  with  model-based  methods.  To  take  advantage  of  model- 
based  techniques  to  develop  systems,  it  is  important  to  improve  human  and  technology 
integration  to  make  trades  and  decide  on  what  is  most  effective  given  the  present  knowledge 
and  future  uncertainties,  as  well  as  make  logical  decisions  based  on  the  availability  of  resources 
and  constraints.  The  Interactive  Model-Centric  Systems  Engineering  (IMCSE)  research  program 
will  develop  the  SE  methods,  processes  and  tools  to  improve  this  interaction,  with  the  goal  of 
accelerating  the  transition  of  SE  to  become  a  more  model-based  discipline. 

The  IMCSE  research  program  aims  to  develop  transformative  results  through  enabling  intense 
human-model  interaction,  to  rapidly  conceive  of  systems  and  interact  with  models  in  order  to 
make  rapid  trades  to  decide  on  what  is  most  effective  given  present  knowledge  and  future 
uncertainties,  as  well  as  what  is  practical  given  resources  and  constraints. 


Work  Accomplished  in  Phase  2 

The  IMCSE  research  program  involves  three  tasks  initiated  in  May  2014,  with  two  additional  tasks 
added  for  Phase  2  that  extended  from  the  original  set  of  three.  The  work  accomplished  in  this 
second  phase  includes: 

1.  Pathfinder  Project.  This  effort  has  continued  to  investigate  the  current  state  of  the  art/practice 
in  IMCSE.  The  research  team  conducted  knowledge  gathering  and  additional  literature  review, 
building  on  the  Phase  1  work.  The  expanded  knowledge  gathering  was  used  to  inform  the  design 
of  an  invited  workshop,  focused  on  identifying  research  opportunities,  gaps  and  issues  along  with 
associated  priorities  and  initial  plans.  The  team  designed,  planned,  and  conducted  a  Pathfinder 
IMCSE  Workshop  on  20  January,  2015.  The  workshop  report  is  included  in  Appendix  A. 

2.  Interactive  Schedule  Reduction  Model  (ISRM).  The  research  team  continued  development  of 
the  ISRM,  which  was  based  on  an  existing  prototype  system  dynamics  (SD)  model  to  interactively 
explore  alternatives  in  the  systems  development  process  and  application  of  resources.  The  model 
enables  rapid  sensitivity  analysis  of  various  factors  to  determine  their  potential  impact  on 
program  schedule,  and  investigates  new  methods  for  human  interaction  with  the  model. 
Extending  the  effort  in  phase  1,  the  team  completed  Phase  2  objectives  for  demonstrating  a 
service-based  tool  to  support  rapid  sensitivity  analysis.  A  back-end  implementation  uses  a 
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MongoDB/Express/Node.js  stack  to  provide  remote  model  execution  and  services  to  store  and 
query  model  results.  Front-end  user  interfaces  support  several  activities.  An  execution  module 
performs  full-factorial  design  of  experiments  to  vary  parameters  of  interest,  execute  a  local  or 
remote  model,  and  store  results.  A  visualization  module  provides  three  capabilities  based  on 
result  query  services:  1)  time  series  comparison  to  visualize  simulation  results  across  several 
scenarios,  2)  sensitivity  analysis  to  compare  relative  magnitude  of  results  to  a  baseline  scenario, 
and  3)  tradespace  exploration  to  visualize  all  results  in  a  two-dimensional  space.  These  features 
exceed  the  existing  capabilities  of  Vensim  and  have  been  demonstrated  to  analyze  and  visualize 
results  across  more  than  1000  scenarios.  The  completed  prototype  provides  a  proof-of-concept 
prototype  that  has  potential  to  inform  future  interactive  model  development. 

3.  Interactive  Epoch-Era  Analysis.  The  research  team  further  developed  a  strategy  for  extending 
a  current  approach  for  evaluating  systems  under  uncertainty,  Epoch-Era  Analysis  (EEA),  through 
the  development  of  an  interactive  capability.  Effort  has  focused  on  the  exploratory  development 
of  Interactive  Epoch-Era  Analysis  (IEEA)  methods,  including  human  interface  and  reasoning 
considerations  for  epoch  and  era  characterizations,  as  well  as  single  and  multi-  epoch  and  era 
analyses.  The  team  has  explored  visualization  techniques  and  methods  for  mitigating 
computational  resource  restrictions  that  facilitate  improved  decision-making.  A  preliminary 
method  has  been  developed  with  a  demonstration  prototypes,  and  case  examples.  Various 
potential  visual  representations  and  user  interaction  flows  were  proposed  during  this  phase,  and 
some  are  being  implemented  for  potential  user  testing.  These  include  epoch  characterization  and 
selection,  and  era  construction  and  analyses.  The  team  has  identified  key  user  tasks  and 
objectives  to  be  further  evaluated  during  user  tests. 

The  following  two  projects  were  added  for  the  Phase  2  period. 

4.  Supporting  MPTs.  During  Phase  2,  the  research  team  continued  to  define  and  prototype 
implementations  of  parts  of  Interactive  Epoch-Era  Analysis. 

5.  Trading  Models.  During  this  phase,  the  research  team  expanded  its  research  on  trading 
models,  with  further  development  of  an  exploratory  case  on  value  model  trading.  The  team 
began  developing  an  alternative  performance  model  for  the  demonstration  case  to  be  added  in 
Phase  3. 


Research  Results 

The  research  team  has  produced  interim  research  outcomes  for  each  of  the  three  research 
thrusts  in  the  project:  foundations,  fundamentals,  and  applications.  These  outcomes  feed 
forward  into  Phase  3  of  the  project.  The  following  findings  have  resulted  from  the  Phase  2  effort 
over  the  5  month  period  of  performance: 

1.  Phase  1  investigation  led  to  the  identification  of  three  challenges  at  the  intersection  of 
the  four  pillars  (key  topic  areas):  tradeoff  of  models,  visual  analytics  of  artificial  data,  and 
perceptual  and  cognitive  considerations  in  human-model  interaction.  In  Phase  2  these 
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have  been  further  explored,  and  specific  research  needs  were  identified.  These  have  been 
incorporated  into  the  Phase  3  research  plan. 

2.  The  Pathfinder  IMCSE  Workshop  was  designed,  planned  and  conducted  on  20  January 
2015.  The  workshop  demonstrated  the  interest  around  the  IMCSE  topic,  and  validated 
the  need  for  much  more  research  in  this  area.  Results  of  the  workshop  have  been 
published  (see  Appendix  A),  and  feed  forward  into  Phase  3  to  further  evolve  the  research 
agenda  and  roadmap,  and  define  priorities. 

3.  The  Interactive  Schedule  Reduction  Model  (ISRM)  was  completed,  demonstrating  web- 
based  technologies  as  new  methods  for  human-model  interaction  enabling  rapid 
sensitivity  analysis  of  various  factors.  The  resulting  proof-of-concept  prototype  with 
supporting  documentation  is  available  as  a  model  for  how  such  technology  can  be  used. 

4.  A  preliminary  method  for  Interactive  Epoch-Era  Analysis  (IEEA)  has  been  developed  with 
demonstration  prototypes,  and  case  examples.  Various  potential  visual  representations 
and  user  interaction  flows  were  proposed  for  potential  use. 

5.  A  demonstration  case  for  value  model  choice  and  tradeoffs  was  refined  and  applied  in  a 
Space  Tug  system,  highlighting  methodological  considerations. 


Next  Steps 

•  The  research  team  will  use  knowledge  from  Phase  1  and  Phase  2  to  focus  ongoing  efforts  in 
Phase  3  to  further  explore  the  identified  IMCSE-related  considerations  within  four  key  areas, 
and  the  challenges  and  opportunities  at  their  intersection. 

•  The  Pathfinder  Workshop  Report  (Appendix  A)  will  be  disseminated  to  elicit  comments  and 
recommendations,  augmented  by  discussions  with  selected  subject  matter  experts.  This  will 
feed  into  creating  a  collaboratively-derived  research  agenda.  A  research  roadmap  will  be 
derived  in  collaboration  with  other  SERC  researchers  and  the  broader  systems  community.  A 
leadership  summit  is  targeted  to  support  validation  of  research  priorities,  recommend 
pathways  to  accelerate  research  progress,  and  enable  transition  to  the  systems  community. 

•  The  research  team  will  mature  the  approach  for  evaluating  systems  under  dynamic 
uncertainty,  with  further  development  of  the  extended  framework  to  for  interactive 
capability  and  scaling  to  big  data.  This  work  extends  the  Phase  2  effort  on  a  demonstration 
prototype,  usingthe  MITIVTea  Suite,  applying  IMCSE  principles  to  enhance  the  user  interface, 
data  handling  and  analysis  widgets.  In  Phase  3  the  research  team  will  enhance  the  method 
and  degree  of  interactive  capability,  focusing  specifically  on  the  Epoch-Era  Analysis  method, 
a  novel  method  for  value-driven  tradespace  exploration  and  analysis.  The  maturing 
prototype  framework  with  associated  supporting  tools  will  be  applied  to  a  case  analysis 
including  various  types  of  uncertainties.  This  case  application  will  be  used  to  elicit  feedback 
on  relevance,  ease  of  use,  feasibility  and  tractability  of  data  scaling  and  visualization 
techniques.  The  research  team  will  extend  the  Phase  2  prototype  efforts  for  Interactive 
Epoch-Era  Analysis  (IEEA)  and  test  using  case  applications,  along  with  preliminary  supporting 
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infrastructure.  This  will  inform  the  transition  strategies,  additional  case  application  and 
prototype  user  testing. 

•  The  research  team  will  build  on  the  Phase  2  work  on  value  model  trades  to  further  evolve  the 
framework  and  process.  In  Phase  3,  the  research  team  will  build  on  prior  phase  results  to 
further  evolve  the  framework  and  process  for  conducting  value  model  choice  and  tradeoffs 
and  apply  this  through  an  expanded  case  application  set,  to  validate  the  framework  and 
identify  workflow  considerations.  The  model  choice  and  tradeoff  framework  will  be  expanded 
including  demonstration  cases  beyond  value  models  (to  include  trading  of  other  types  of 
models  including  performance  and  cost  models).  The  expanded  framework  will  consider 
alternative  use  cases  for  the  impact  of  model  choice  and  tradeoffs  on  decision-making.  For 
example,  this  includes  the  context  of  multi-stakeholder  negotiations  using  tradespace 
exploration,  where  the  data  source(s)  (i.e.  "models")  strongly  impact  the  trust  and  framing  of 
the  shared  decision  problem. 

•  The  research  team  will  continue  to  investigate  the  cognitive  and  perceptual  considerations  in 
human-model  interaction,  a  topic  for  which  little  research  exists,  though  there  is  a  body  of 
knowledge  to  draw  from.  Preliminary  heuristics/design  principles  will  be  gathered,  adapted 
for  human-model  applicability,  and  synthesized  as  a  draft  guidance  document.  The  guide  will 
be  shared  for  review  and  comment  by  model  developers,  users  and  model-based  software 
designers,  toward  publication  of  a  validated  set  of  guiding  principles  for  effective  human- 
model  interaction  during  Phase  4.  A  goal  is  to  involve  one  or  more  SERC  collaborators  as 
transition  partners,  to  pilot  use  of  the  guiding  principles  during  Phase  3. 

•  The  research  team  will  use  the  results  of  Phase  1  and  Phase  2,  along  with  ongoing  Phase  3 
research  interim  results,  to  develop  several  publishable  papers  for  journal  and  conference 
submissions.  Evolving  prototype  MPTs  will  be  shared  and  demonstrated  at  one  or  more  SERC- 
related  events  during  Phase  3,  including  the  CSER  2015.  The  research  team  will  continue 
active  knowledge  exchanges  with  several  other  SERC  researchers  performing  related  work, 
where  IMCSE  outcomes  can  inform  and/or  be  applied  in  their  work. 


li 


Year 


Focus 


Key  Deliverables 


Pre-2014  New  start 

Pathfinder  project  with  collaborative 
research  discovery;  exploratory 
extensions  to  an  existing  development 
schedule  reduction  model;  exploratory 
piloting  and  interactive  extensions  to 
Epoch-Era  method 

Pathfinder  Project:  Investigation  of  current  state  of 
art/practice.  Workshop  to  explore  issues  and 
opportunities,  with  report  on  workshop  results. 
Pathfinder  project  report,  with  findings  of  research 
opportunities,  gaps,  and  issues.  Out-year  research 
plans  based  on  pathfinder  results. 

Interactive  Schedule  Reduction  Model:  Exploratory 
extensions  implemented  and  evaluated.  Report  on 
exploratory  schedule  model.  Prototype  model  for  pilot 
application. 

Interactive  Epoch-Era  Analysis:  Exploratory  research  to 
develop  interactive  capability,  with  demonstration  via  a 
mission  planning  support  application  case.  Report  on 
exploratory  research  and  case  application. 

Initiate  multi-year  research  plans  based 
on  pathfinder  results,  including  2014 

2015  project  follow-on  for  one  or  both  of  the 

exploratory  research  projects.  Assess 
results  individually  and  comparatively. 

IMCSE  Project  Applications:  Based  on  pathfinder  project 
results,  select  and  initiate  one  or  more  additional 
projects,  and  increase  SERC  member  collaboration  in 
projects.  Report  to  document  the  maturation  of  the 
MPTs  for  each  of  these  projects,  with  comparative 
results. 

Increasing  maturation  of  IMCSE  MPTs 

and  enabling  environments,  leading  to 

adoption  by  user  community  and 

assessment  of  real-world  impact; 

extend  IMCSE  scope  via  increased 

2016  K 

collaboration  of  additional  universities 

and  broader  user  community. 

Exploration  of  further  new-idea 

projects. 

IMCSE  MPT  Implementations  and  Impact  Assessments: 
Continued  maturation  and  implementation  of  IMCSE 
MPTs,  with  enabling  environments.  Ongoing  study  of 
impacts  resulting  in  a  comprehensive  report  of 
progress,  results,  and  opportunities. 

Increasing  maturation  and  synthesis  of 
IMCSE  MPTs  and  enabling 
environments,  leading  to  adoption  by 
user  community  and  demonstration  of 

2017-2018 

real-world  impact;  sustain  and  increase 
collaboration  of  additional  universities 
and  broader  user  community.  Step-ups 
of  new-idea  projects. 

IMCSE  MPT  Synthesis  Impact  and  Effective  Practice 
Assessments:  Continued  maturation,  synthesis  and 
implementation  of  IMCSE  MPTs,  with  enabling 
environments.  Ongoing  study  of  real-world  impacts  to 
identify  successful  practices.  A  comprehensive  report  of 
impacts  and  insights,  with  guidance  on  practice. 

Figure  1.  IMCSE  Project  Timeline 
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Introduction 


The  IMCSE  research  program  aims  to  develop  transformative  results  through  enabling  intense 
human-model  interaction,  to  rapidly  conceive  of  systems  and  interact  with  models  in  order  to 
make  rapid  trades  to  decide  on  what  is  most  effective  given  present  knowledge  and  future 
uncertainties,  as  well  as  what  is  practical  given  resources  and  constraints. 


Motivation 

Models  have  significantly  changed  systems  engineering  practice  over  the  past  decade.  Most 
notably,  model-based  systems  engineering  (MBSE)  methods  and  tools  are  increasingly  used 
throughout  the  entire  system  lifecycle  to  generate  systems,  software  and  hardware  products, 
replacing  labor-intensive  and  error-prone  documentation-based  processes  with  model-based 
ones.  While  substantial  benefits  have  been  achieved,  the  most  impactful  application  of  models 
in  systems  engineering  has  yet  to  be  realized.  Models  are  needed  to  inform  engineering 
decisions.  Truly  transformative  results  will  only  come  through  intense  human-model  interaction, 
to  rapidly  conceive  of  systems  and  interact  with  models  in  order  to  make  rapid  trades  to  decide 
on  what  is  most  effective  given  present  knowledge  and  future  uncertainties,  as  well  as  what  is 
practical  given  resources  and  constraints. 

As  cited  in  the  SERC  2014-2018  Technical  Plan,  reports  have  found  significant  insufficiencies  in 
the  current  practice. 

The  National  Research  Council’s  “Human-System  Integration  in  the  System 
Development  Process,  ”  (NRC,  2007),  “Pre-Milestone  A  and  Earlv-Phase  SE,  ” 

(NRC,  2008),  and  “Critical  Code,  ”  (NRC,  2010)  studies  consistently  found  that  the 
SE  MPTs  for  integrating  hardware  engineering,  human  factors  engineering,  and 
software  engineering  into  a  scalable,  unified  approach  were  not  up  to  the 
challenges  of  the  complexity,  scale,  and  dynamism  characterizing  DoD ’s  large- 
scale  systems  and  systems  of  systems. 

This  research  project  addresses  the  SERC's  Systems  Engineering  and  Systems  Management 
Transformation  (SEMT)  grand  challenge: 

Move  the  DoD  community ’s  current  systems  engineering  and  management  MPTs 
and  practices  away  from  sequential,  single  stovepipe  system,  hardware-first, 
outside-in,  document-driven,  point-solution,  acquisition-oriented  approaches; 
toward  concurrent,  portfolio  and  enterprise-oriented,  hardware-software-human 
engineered,  balanced  outside-in  and  inside-out,  model-driven,  set-based,  full  life 
cycle  approaches.  These  will  enable  much  more  rapid,  concurrent,  flexible, 
scalable  definition  and  analysis  of  the  increasingly  complex,  dynamic,  multi¬ 
stakeholder,  cyber-physical-human  DoD  systems,  systems  of  systems,  portfolios  of 
systems,  and  enterprises  of  the  future. 


13 


Insufficiencies  in  Current  Practice 


Early  concept  decisions  have  always  been  critically  important,  and  with  continuously  evolving 
systems  of  systems  having  long  life  spans,  such  decisions  are  now  made  throughout  the  entire 
life  cycle.  Soft  factors  become  increasingly  influential.  For  example,  trust  in  model-based  data 
sets  and  decisions  are  in  part  determined  by  the  chosen  model  itself  as  perceived  by  specific 
decision  makers.  The  timescale  of  making  early  architectural  decisions  is  out  of  sync  with  the 
current  model-based  systems  engineering  capabilities  and  decision  environments.  New 
algorithms  and  novel  modeling  approaches  must  be  discovered  to  accelerate  technical  and 
programmatic  decision  support  from  months  to  minutes.  In  order  to  effectively  leverage  and 
incorporate  human  knowledge  and  judgment,  an  interactive  capability  is  needed.  Much 
potential  exists  in  maturing  emerging  novel  methods  for  evaluating  system  responsiveness  under 
complex  uncertainties,  to  enable  engineering  of  resilient  systems. 

The  Phase  2  Pathfinder  Workshop  Report  (Appendix  A)  includes  further  discussion  on  these 
insufficiencies,  as  identified  by  participants  in  the  workshop. 


Relevant  Prior  SERC  Research 

IMCSE  will  include  and  significantly  extend  the  traditional  focus  on  the  modeling  of  system 
products  and  the  use  of  the  models.  Extensions  will  address  the  modeling  of  system  execution 
processes,  such  as  operational  concept  formulation,  and  system  development  processes,  which 
can  also  be  executed  to  aid  in  the  generation  of  system  products.  As  emphasized  in  the  SERC 
Systems  2020  Report,  an  additional  focus  on  modeling  the  system's  environment  will  be  pursued, 
which  is  needed  for  performing  many  of  the  ilities  tradespace  and  affordability  analyses.  Models 
can  also  improve  affordability  by  automatically  generating  needed  documentation,  or  even 
better  by  serving  as  the  documentation  itself.  Further,  models  can  reduce  or  avoid  system 
overruns  and  performance  shortfalls  by  enabling  more  thorough  Analyses  of  Alternatives  and 
evidence-based  decision  reviews.  Modeling  the  system's  dynamic  operational  environment 
remains  an  open  area  of  research.  IMCSE  has  a  relationship  to  many  of  the  past  and  ongoing 
SERC  projects.  Several  of  the  most  relevant  prior  SERC  projects  are  summarized  in  the  previously 
published  SERC  IMCSE  Phase  1  report. 
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IMCSE 


Interactive  Model-centric  Systems  Engineering  (IMCSE),  not  to  be  confused  with  Model-based 
Systems  Engineering  (MBSE),  is  a  research  program  that  seeks  to  encourage  the  development  of 
augmented  complex  systems  thinking  and  analysis  to  support  data-driven  decision  making. 


What  is  IMCSE? 

Systems  scientists  have  long  recognized  that  humans  possess  unique  abilities  for  anticipation 
rather  than  simple  reactive  response.  In  order  to  increase  the  likelihood  of  developing  complex 
systems  that  can  deliver  value  to  stakeholders  across  a  dynamic,  uncertain  future,  systems 
engineers  must  have  both  reactionary  and  anticipatory  capacity  to  make  better  decisions.  In 
contrast  to  reactionary  capacity,  which  involves  developing  solutions  after  the  fact,  anticipatory 
capacity,  as  defined  by  Rhodes  and  Ross  (2009),  is  "the  capacity  to  continuously  develop  and 
apply  knowledge  acquired  through  a  structured  approach  to  anticipate  1)  changing  scenarios  as 
stakeholder  needs  and  systems  context  change  over  time;  2)  to  consider  their  consequences;  and 
3)  to  formulate  design  decisions  in  response1.  Three  key  enablers  of  anticipatory  capacity  are 
mindset,  methods,  and  environment.  Models  represent  an  abstraction  of  reality  in  order  to  make 
predictions  about  the  future.  Models  can  come  in  a  variety  of  forms  and  formats,  but 
fundamentally  they  are  an  encapsulation  of  reality  that  humans  use  to  augment  their  ability  to 
make  sense  of  the  world  and  anticipate  future  outcomes.  Improvements  in  computation, 
simulation  technologies,  and  human-machine  interaction  have  created  an  opportunity  to  enable 
human-model  interaction  to  greatly  enhance  anticipatory  capacity.  Complex,  integrated  models, 
of  various  levels  of  fidelity,  can  create  large  data  sets  in  need  of  human  pattern  recognition  skills. 
Interaction  enables  real  time  interrogation  of  the  data  and  opportunities  for  model  creation  as 
well  as  validation  and  learning.  IMCSE  is  a  research  program  intended  to  leverage  human-model 
interaction  in  order  to  transform  systems  engineering  decision  making  through  anticipatory 
capacity. 


Research  Program  Vision 

The  vision  for  the  IMCSE  research  program  is  to  develop  transformative  results  through  enabling 
intense  human-model  interaction,  to  rapidly  conceive  of  systems  and  interact  with  models  in 
order  to  make  rapid  trades  to  decide  on  what  is  most  effective  given  present  knowledge  and 
future  uncertainties,  as  well  as  what  is  practical  given  resources  and  constraints. 

In  order  to  accomplish  this  vision,  IMCSE  will  pursue  a  balanced  basic  and  applied  research 
approach.  This  will  leverage  the  strength  of  the  academic  environment  (e.g.  developing 
fundamentals,  approaching  with  rigor,  providing  a  neutral  third  party  view  of  the  problem). 
Additionally,  IMCSE  will  strive  to  keep  the  research  relevant  to  the  sponsor  community,  as  well 
as  enabling  opportunities  for  knowledge  and  methods,  processes,  and  tools  (MPTs)  transfer  to 


1  Rhodes,  D.H.  and  Ross,  A.M.,  "Anticipatory  Capacity:  Leveraging  Model-Based  Approaches  to  Design  Systems  for 
Dynamic  Futures,"  2nd  Annual  Conference  on  Model-based  Systems,  Haifa,  Israel,  March  2009. 
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sponsors.  Such  knowledge  transfer  opportunities  include  workshops,  teleconferences  and 
meetings,  reports,  papers,  collaboration  with  other  SERC  activities,  prototypes,  methods, 
processes,  and  tools  (MPTS),  government  partner  applications,  and  potential  student  internships. 


IMCSE  Pillars  -  Four  Topic  Areas 

IMCSE  is  motivated  by  the  convergence  of  four  key  topic  areas:  big  data,  visual  analytics,  complex 
systems,  and  model-based  systems  engineering.  Each  of  these  areas  have  associated  with  them 
large  research  and  application  efforts.  This  research  program  seeks  to  identify  synergies  and 
gaps  at  the  intersection  of  these  four  topic  areas,  and  leverage  existing  and  new  techniques  in 
this  area  to  create  new  knowledge  and  capabilities  for  systems  engineering  decision  making.  In 
order  to  focus  the  research  program,  early  efforts  are  aimed  to  identify  key  challenges  that 
summaries  the  gaps  in  the  existing  topic  area  overlaps. 


Big  Data 

We  live  in  a  world  with  big  data.  As  data  storage  costs  have  shrunk,  so  too  has  the  need  for 
purging  data.  Additionally  data  is  being  generated  through  a  large  and  growing  number  of 
means,  from  sensors  to  users  to  corporate  IT  environments.  Even  "document-based"  data  is 
becoming  digital  as  technology  (including  OCR)  becomes  commonplace  for  capturing  physical 
information  as  digital  data.  No  consensus  currently  exists  regarding  a  formal  definition  on  what 
constitutes  "big  data,"  but  it  is  generally  recognized  as  having  a  number  of  characteristics  that 
make  it  "big."  One  example  description,  from  IBM2,  characterizes  big  data  as  having  challenges 
regarding  Volume,  Variety,  Velocity,  and  Veracity.  The  challenge  for  Volume  revolves  around  the 
scale  of  the  data  (e.g.  how  to  store  and  recall  large  numbers  of  field  entries  in  a  database?).  The 
challenge  for  Variety  revolves  around  the  different  forms  of  data  (e.g.  how  to  store  and  compare 
data  from  photos,  videos,  blogs,  articles,  etc.?).  The  challenge  for  Velocity  revolves  around  the 
analysis  of  streaming  data  (e.g.  how  to  account  for  and  parse  large  streams  of  potentially 
incomplete  data  in  real  time?).  The  challenge  for  Veracity  revolves  around  the  uncertainty  of  the 
data  (e.g.  different  data  sources  have  different  degrees  of  trustfulness  and  reliability,  so  how  to 
fuse  data  from  such  sources?). 

The  impact  of  big  data  is  being  felt  across  many  fields  from  transportation  to  entertainment, 
education  to  banking,  which  will  only  increase  as  the  benefit  of  leveraging  such  data  becomes 
apparent.  Such  benefits  have  been  recognized  by  a  growing  number  of  commercial  organizations 
who  are  leveraging  this  inundation  of  data  to  gain  insights  into  phenomena  to  create  predictive 
models  (e.g.  of  user  behavior  and  preferences).  For  example,  Amazon  and  Netflix  both  have 
sophisticated  user  preference  models  that  are  used  to  make  recommendations  to  users  based 
on  their  own  (and  related  others)  browsing  and  shopping/viewing  history.  Additionally,  Netflix 
has  used  this  information  (and  Amazon  recently  as  well)  to  generate  design  requirements  for 


2  http://www.ibmbigdatahub.com/sites/default/files/infographic  file/4-Vs-of-big-data.jpg 
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new  shows.  House  of  Cards,  produced  by  Netflix,  was  partially  designed  based  on  derived 
preferences  of  its  viewer  base  in  order  to  increase  the  perceived  value  of  the  program3. 

While  not  necessarily  generated  in  a  similar  manner,  DoD  has  already  a  vast  amount  of  data 
stored  in  documents,  for  example  requirements  documents,  design  documents,  DoDAF,  etc, 
which  represent  latent  data  that  could  be  leveraged  using  techniques  being  developed  in  the 
commercial  application  space.  What  would  a  ground  vehicle  recommendation  look  like?  How 
would  it  parse  and  analyze  historical  requirements  documents  and  contextual  information  in 
order  to  predict  and/or  augment  modern  user  needs?  Big  data  is  a  topic  area  that  holds  promise 
in  providing  a  foundation  for  large  scale  analytics  to  predict  the  future. 


Visual  Analytics 

Visual  analytics  is  a  topic  area  that  has  likewise  been  a  growing  area  for  research  and  application. 
At  its  core,  visual  analytics  is  about  collaboration  between  human  and  computer  using 
visualization,  data  analytics,  and  human-in-the-loop  interaction.  More  than  just  visualization 
tools,  visual  analytics  aims  to  take  advantage  of  a  human's  ability  to  discover  patterns  and  drive 
inquiry  in  order  to  make  sense  of  data.  In  2007,  DHS  sponsored  the  National  Visualization  and 
Analytics  Center,  which  developed  a  research  agenda  called  Illuminating  the  Path.  In  it,  visual 
analytics  was  defined  as  "the  science  of  analytical  reasoning  facilitated  by  interactive  visual 
interfaces"  that  "provides  the  last  12  inches  between  the  masses  of  information  and  the  human 
mind  to  make  decisions."4  Application  areas  range  from  homeland  security  to  anti-fraud,  banking 
to  insurance.  One  common  element  in  much  of  the  current  visual  analytics  work  involves  case 
applications  comparing  VA-supported  inquiry  results  to  ground  truth,  that  is,  discovery  of 
patterns  in  "natural"  data.  One  consequence  of  these  studies  is  that  the  validity  of  the 
applications  can  be  compared  to  observable  "truth."  This  allows  researchers  to  test  how  well 
their  predictive  models  match  reality,  for  example,  using  VA  to  discover  hackers  trying  to  break 
into  streams  of  ATM  data;  or  discovering  patterns  of  use  in  bike  sharing  programs  as  a  function 
of  time  and  geography.  In  both  of  these  examples  there  are  "real"  processes  at  play  and  actual 
measurable  real  world  data  against  which  to  validate  predictions  by  the  human-machine  VA 
system.  VA  has  been  shown  to  be  incredibly  useful  for  developing  models  of  natural  data. 


Complex  Systems 

Our  application  domain  is  the  development  of  (artificial)  systems  that  serve  the  purpose  of 
delivering  value  to  stakeholders.  By  "artificial"  we  mean  that  these  systems  are  artifacts  created 
by  humans  for  a  purpose,  to  be  contrasted  with  natural  systems,  which  are  not  created  by 
humans.  Over  time,  the  complexity  of  systems  has  tended  to  grow,  not  only  due  to  scale  and 
interconnectedness,  but  also  due  to  increased  scope  in  our  ability  to  describe  the  system.  This 
enhanced  scope  reflects  realization  that  the  success  of  artificial  systems  requires  a  fuller 


3http:/A/vww.  nytimes.com/2013/02/25/business/media/for-house-of-cards-using-big-data-to-guarantee-its- 

popularity.html?pagewanted=all&  r=0 

4  Jim  Thomas,  Director,  USDHS  National  Visualization  and  Analytics  Center,  "Visual  Analytics:  An  Agenda  in  Response 
to  DHS  Mission  Needs,"  2007 
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understanding  of  how  the  system  is  structured,  behaves,  performs  in  different  contexts, 
performs  over  time,  is  perceived  across  stakeholders5  6.  This  means  that  to  describe  a  complex 
system,  one  must  consider  all  five  perspectives,  thereby  creating  a  richer  description  of  the 
system.  Developing  complex  systems  necessitates  an  approach  to  generate,  manage,  and 
analyze  artificial  data  across  these  five  aspects. 


Model-Based  Systems  Engineering  (MBSE) 

Traditional  systems  engineering  has  been  document-heavy  and  process-driven,  resulting  in  many 
opportunities  for  miscommunication  and  mistakes  during  "hand-offs"  between  phases  and 
teams.  Models  are  often  used  during  design  and  development  in  order  to  predict  behavior  or 
other  consequences  of  design  decisions,  before  the  system  is  built  or  operated.  In  contrast  to 
document-based  engineering,  "model-based  systems  engineering  (MBSE)  is  the  formalized 
application  of  modeling  to  support  system  requirements,  design,  analysis,  verification,  and 
validation  activities  beginning  in  the  conceptual  design  phase  and  continuing  throughout 
development  and  later  life  cycle  phases."5 6 7  Today,  however,  standalone  models  are  typically 
related  through  documents.  A  future  vision  is  for  organizations  to  use  "shared  system  model(s) 
with  multiple  views,  and  connected  to  discipline  models,"  in  order  to  reduce  effort  creating  and 
aligning  documents,  and  to  increase  synthesis  and  coherence  across  disciplines  throughout 
design8.  Regardless  of  the  degree  to  which  MBSE  is  employed,  its  benefits  stem  from  moving  to 
models  to  represent  systems  with  less  ambiguity,  more  parsimony,  and  more  consistency, 
resulting  in  reduced  acquisition  time,  enhanced  reliability,  etc.  MBSE  generates  "artificial  data" 
about  systems  which  can  be  used  to  make  decisions  that  impact  the  future  and  continuing 
success  of  that  system. 


Synthesizing  the  Pillars 

Each  of  the  four  topic  areas  above  are  themselves  large  areas  of  active  research  and  development 
across  government,  academia,  and  industry.  IMCSE  in  particular  is  interested  in  the  intersection 
of  these  four  areas  with  application  to  improving  systems  engineering  decision  making.  More 
than  just  applied  visual  analytics,  IMCSE  seeks  to  look  at  data  generated  by  models,  in  order  to 
make  better  decisions  in  how  to  deliver  sustained  value  to  stakeholders.  In  particular  preliminary 
investigation  has  uncovered  two  initial  challenges  for  IMCSE  to  address. 

These  include: 

1)  Visual  analytics  of  artificial  (i.e.  model-generated)  data:  how  does  this  differ  from  VA  of 
natural  data?  How  to  take  into  account  the  impact  of  various  model  implementations  on 


5  Rhodes,  D.H.,  and  Ross,  A.M.,  "Five  Aspects  of  Engineering  Complex  Systems:  Emerging  Constructs  and  Methods," 
4th  Annual  IEEE  Systems  Conference,  San  Diego,  CA,  April  2010 

6  Gaspar,  H.,  Rhodes,  D.H.,  Ross,  A.M.,  and  Erikstad,  E.O.,  "Addressing  Complexity  Aspects  in  Conceptual  Ship  Design: 
A  Systems  Engineering  Approach"  Journal  of  Ship  Production  and  Design,  Vol.  28,  No.  4,  Nov  2012,  pp.  145-159. 

7 INCOSE  SE  Vision  2020  (INCOSE-TP-2004-004-02,  Sep  2007) 

8  http://www.omgwiki.org/MBSE/doku.php?id=mbse:incose_mbse_iw_2014 
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pattern  finding  and  matching  of  mental  and  constructed  models?  How  to  validate 
predictions  without  ground  truth  available? 

2)  Active  tradeoffs  of  models  themselves:  too  often  models  are  used  without  sufficient 
investigation  into  the  impact  of  the  models  on  the  data  being  used  for  decisions;  these 
include  performance  models,  cost  models,  and  value  models.  Model  selection 
fundamentally  impacts  the  patterns  to  be  discovered  in  the  artificial  data. 

Ultimately,  the  goal  of  IMCSE  is  to  leverage  visual  analytics  applied  to  model-generated  "big 
data,"  in  order  to  develop  a  rigorous  framework,  with  associated  methods ,  processes,  and  tools 
(MPTS),  which  will  result  in  transformative  new  capabilities  for  complex  systems  engineering 
decision  making. 


IMCSE  Approach 

IMCSE  uses  three  complimentary  thrusts  with  different  timescales,  in  order  to  have  impact  on 
the  long  term,  the  near  term,  and  the  present. 

These  thrusts  include: 

•  Foundations:  1  year,  set  the  stage  for  IMCSE  for  long  term  impact 

•  Fundamentals:  multi-year,  medium  timescale  impact,  potentially  broad  applicability 

•  Applications:  1  year,  short  timescale  impact,  generate  deployment  opportunities 

Current  progress  in  each  of  these  three  thrusts  will  be  described  in  the  following  sections  of  the 
report. 


19 


Foundations 


The  foundations  research  thrust  is  currently  focused  on  two  activities.  The  first  is  the  research 
pathfinder  project,  including  the  initial  'setting  the  stage'  activity  of  an  invited  workshop,  which 
was  conducted  during  Phase  2.  Extending  the  results  of  the  workshop,  a  more  extensive  effort  in 
Phase  3  will  build  a  community  of  interest.  The  end  result  will  be  a  collaboratively-derived 
research  roadmap  and  set  of  priorities. 

The  second  activity  is  investigating  the  current  state  practice  and  emerging  state  of  the  art.  This 
includes  literature  review  and  discussions  with  subject  matter  experts.  Ongoing  results  continue 
to  inform  the  research  agenda,  and  specific  projects  undertaken  in  the  fundamentals  and 
applications  thrusts. 


IMCSE  Research  Pathfinder  Project 

A  pathfinder  project  brings  together  the  relevant  stakeholders  to  develop  a  research  vision  and 
research  priorities,  and  a  roadmap  to  achieve  them.  The  IMCSE  pathfinder  project  includes  face- 
to-face  gatherings  of  stakeholders  in  the  research  agenda,  as  well  as  specifically  focused  research 
meetings.  An  initial  workshop  held  during  Phase  2  has  seeded  the  initial  research  agenda.  Given 
the  footprint  of  IMCSE,  it  would  not  be  possible  to  convene  a  large  enough  community  in  a 
participant  workshop  for  the  purpose  of  a  collaboratively-derived  research  agenda.  Our  research 
team  has  explored  various  approaches  that  have  been  used,  and  will  leverage  the  ideas  from 
these  approaches.  The  goal  is  to  be  able  to  engage  a  large  and  diverse  community  around  the 
research  agenda,  and  determine  an  approach  that  may  include  both  face-to-face  and  virtual 
activities. 


Pathfinder  Project 

Preliminary  efforts  in  defining  a  research  vision  and  results  of  exploratory  knowledge  gathering 
have  been  used  to  design  and  conduct  the  IMCSE  Pathfinder  Workshop  during  Phase  2.  The 
workshop  was  conducted  on  20  January  2015,  bringing  together  interested  stakeholders  for  an 
initial  dialogue  on  human-model  interaction,  identifying  research  needs  from  both  a  model¬ 
centric  perspective  and  an  interactive  perspective.  A  rich  set  of  participant  observations,  insights, 
feedback  and  recommendations  was  gathered  during  the  event.  Stage-setting  talks  were 
employed  to  orient  the  participants  on  the  four  "pillars"  and  some  of  the  identified  challenges  at 
their  intersection.  Following  the  workshop,  an  activity  has  been  initiated  to  gather  input  from 
SERC  members  and  the  broader  systems  community  to  evolve  a  research  agenda  for  IMCSE.  The 
ultimate  goal  is  to  establish  a  shared  set  of  IMCSE  research  priorities  and  roadmap,  excite  the 
research  community  around  the  topic,  and  build  partnerships  for  research  collaboration  and  for 
transition  to  practice. 
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The  outcome  of  the  workshop  event  is  a  workshop  report  (Appendix  A).  The  event  identified  level 
research  needs  and  questions,  along  with  identifying  gaps,  issues  and  opportunities.  A  vision  for 
the  individual's  experience  in  the  ideal  work  was  captured  during  the  workshop.  The  results  of 
the  workshop  have  been  made  available  for  review  and  comment.  The  goal  of  the  workshop  and 
resulting  report  was  to  seed  the  larger  effort  to  build  a  community  of  interest  and  undertake  a 
more  extensive  research  pathfinder  activity.  Feedback  from  the  review  of  the  report  and  follow- 
on  discussions  with  subject  matter  experts  will  be  captured  and  evolved  during  Phase  3. 

The  pathfinder  activities  in  Phase  1  and  Phase  2  of  this  research  project  inform  efforts  in  Phase  3 
to  elicit  information  on  state  of  the  art  and  practice,  identify  additional  research  stakeholders, 
clarify  and  expand  the  urgent  research  questions,  and  investigate  priorities.  The  Phase  3  objective 
will  be  to  establish  a  collaboratively-derived  IMCSE  research  roadmap.  The  ultimate  goal  is  to 
build  a  community  of  interest  around  the  IMCSE  research  agenda,  build  partnerships  for 
research,  and  to  foster  collaboration  in  addressing  the  emerging  challenges  at  the  intersection  of 
the  four  topic  areas. 


Building  a  Community  of  Interest 

Each  of  the  four  topic  areas  (big  data,  visual  analytics,  complex  systems,  and  model-based 
systems  engineering)  engages  researchers  from  multiple  disciplines  and  domains.  IMCSE 
research  seeks  to  encourage  the  development  of  augmented  complex  systems  thinking  and 
analysis  to  support  data-driven  decision  making.  The  stakeholders  who  contribute  to  and  benefit 
from  IMCSE  include  government  sponsors,  senior  decision  makers,  system  designers,  analysts, 
academic  researchers,  policy  makers,  funding  agencies  and  others.  Bringing  such  a  community 
together  around  a  shared  research  vision  and  agenda  is  a  significant  challenge,  but  there  are  prior 
exemplars. 

The  research  team  has  been  investigating  successful  efforts  in  other  fields  to  create  a  research 
agenda  through  a  collaborative  approach  involving  a  large  and  diverse  set  of  participants.  A 
particular  feature  of  these  exemplars  is  the  success  in  bringing  together  stakeholder  from 
government,  non-governmental  organizations,  academia  and  industry  to  narrow  the  gap 
between  data  generated  in  research  and  the  information  required  by  policy  makers.  One  recent 
effort  was  the  development  of  a  collaboratively-derived  science-policy  research  agenda,  driven 
by  the  need  for  policy  makers  to  understand  science  and  for  scientists  to  understand  policy 
processes.9  Participants  were  selected  to  cover  a  wide  range  of  disciplines  and  constituencies. 
Each  participant  submitted  a  list  of  questions,  resulting  in  239  questions  in  the  first  stage.  A 
process  of  voting,  deliberation  and  further  voting  resulted  in  a  final  set  of  40  questions,  and  then 
grouped  thematically  into  six  groups.  The  authors,  Sutherland  et  al.  (2012),  noted  the  outcome 
is  inevitably  influenced  by  the  composition  of  the  participants  and  the  process.  While  not 
'reproducible,  it  is  highly  likely  this  approach  would  yield  similar  emergent  general  themes. 


9  Sutherland,  W.,  et  al.,  "A  Collaboratively-Derived  Science-Policy  research  agenda",  PLoS  ONE,  Vol.  7,  Issue  3,  March 
2012 
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Ingram,  et  al.  (2013)10  applied  the  collaboratively-derived  research  approach  of  Sutherland  et  al. 
(2012)  in  addressing  questions  related  to  the  UK  food  security  and  food  system.  They  found  it 
proved  useful  for  engaging  a  wide  range  of  stakeholders  and  helped  establish  a  well-balanced 
discussion  on  the  production  system  and  the  security  outcomes,  informing  a  research  agenda 
from  public  funders  and  applied  industry  viewpoints,  as  well  as  mapping  needs  onto  the 
international  food  security  agenda.  The  dimensions  in  this  work  are  not  unlike  IMCSE,  where 
there  are  intertwined  needs  of  defense  funding  agencies,  system  developers,  and  impacts  to 
national/international  security. 

Sutherland,  et  al.  (2011)* 11  discuss  methods  that  "maximize  inclusiveness  and  rigour  in  such 
exercises  include  solicitation  of  questions  and  priorities  from  an  extensive  community,  online 
collation  of  material,  repeated  voting  and  engagement  with  policy  networks  to  foster  uptake  and 
application  of  the  results".  These  authors  summarize  eight  exercises  with  variation  on  the 
general  approach.  Their  work  in  bridging  the  gap  between  scientific  researchers  and  policy 
makers  is  notable,  as  IMCSE  needs  to  bridge  the  gap  between  the  engineer/scientists  and  the 
senior  decision  makers  and  policy  makers.  While  the  authors  work  in  a  different  domain  of 
interest,  their  work  has  resulted  in  a  set  of  guiding  principles  for  generating  a  collaboratively- 
derived  research  agenda.  The  guidance  covers  defining  the  project;  organizing  the  participants; 
soliciting  and  managing  questions  or  issues;  voting  systems;  and  disseminating  results. 
Experiences  and  effective  practices  in  establishing  collaboratively-derived  research  agendas 
provide  insight  and  guidance,  with  potential  to  enhance  the  success  of  the  pathfinder  project  for 
IMCSE. 

In  Phase  2,  the  research  team  worked  toward  a  concept  for  generating  a  collaboratively-derived 
research  agenda,  to  build  on  the  outcomes  of  the  initial  pathfinder  workshop.  In  Phase  3  this  will 
be  developed  into  a  structured  approach  and  implemented  using  some  web-based  technology 
support. 


Exploring  the  IMCSE-relevantState  of  the  Art  and  Practice 

In  Phase  1,  the  research  team  initiated  its  literature  review  and  knowledge  gathering  to  explore 
the  state  of  the  art  and  practice,  specifically  as  related  to  the  IMCSE  area.  The  team's  organizing 
framework  for  investigation  is  around  four  key  topic  areas,  as  well  as  the  emerging  challenges  at 
their  intersections.  The  four  topic  areas  are  (1)  big  data,  (2)  visual  analytics,  (3)  complex  systems 
and  (4)  model-based  systems  engineering.  These  four  areas  have  an  extensive  and  expanding 
landscape,  and  the  goal  of  our  research  is  not  to  establish  a  comprehensive  state  of  the  art  and 
practice  of  the  topic  areas,  but  rather  to  discover  the  critical  themes,  challenges  and  questions 
that  are  directly  relevant  for  IMCSE. 


10  Ingram,  J.,  "Priority  research  questions  for  the  UK  food  system",  Food  Sec.,  (2013)  5:617-636. 

11  Sutherland,  W.  J.,  Fleishman,  E.,  Mascia,  M.  B.,  Pretty,  J.  and  Rudd,  M.  A.  (2011),  Methods  for  collaboratively 
identifying  research  priorities  and  emerging  issues  in  science  and  policy.  Methods  in  Ecology  and  Evolution,  2:  238- 
247 
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During  Phase  1,  three  challenges  at  the  intersections  emerged:  (1)  tradeoff  of  models;  (2)  visual 
analytics  of  artificial  (model-generated)  data;  and  (3)  perceptual  and  cognitive  considerations  in 
human-model  interaction.  These  informed  the  pathfinder  workshop  activity 

These  challenges  were  further  investigated  during  Phase  2,  and  have  evolved  into  specific 
activities.  Research  on  the  tradeoff  of  models  in  Phase  2  has  been  on  value  models.  In  Phase  3 
this  will  be  extended  to  cost  and  performance  models.  The  visual  analytics  of  artificial  data  has 
become  part  of  the  Interactive  Epoch-Era  Analysis  research,  which  will  continue  in  Phase  3. 
Further  investigation  of  the  perceptual  and  cognitive  considerations  during  Phase  2  has  elevated 
the  importance  of  this  topic.  Phase  2  included  literature  review,  presentation  at  conferences,  and 
discussions  with  subject  matter  experts.  Phase  3  will  leverage  the  work  toward  development  of 
heuristic  guidelines  for  human-model  interaction  considerations. 

In  the  following  subsections,  we  highlight  several  of  the  themes  within  each  of  the  four  topic 
areas  and  the  three  intersection  topics. 


Big  Data 

Big  data  provides  a  foundation  for  large  scale  analytics  to  predict  the  future. 

Big  data  provides  a  foundation  for  large-scale  analytics  to  predict  the  future  across  domains  as 
diverse  as  defense,  healthcare,  and  urban  planning.  Yet  as  evolving  technological  capabilities 
allow  for  the  capture,  management,  and  exploration  of  increasingly  large  and  complex  data  sets, 
researchers  are  also  faced  with  new  and  emerging  methodological  questions  when  grappling 
with  the  forecasting  implications  of  big  data.  Broadly  speaking,  two  of  the  most  significant  issues 
facing  researchers  in  the  field  of  big  data  today  are  trust  in  the  data  and  representativeness  of 
the  models  they  engender. 

Overprojection  of  trend  models.  Perhaps  one  of  the  clearest  examples  highlighting  both  the 
promises  and  pitfalls  of  big  data  driven  analytics  is  the  recent  Google  Flu  Trends  (GFT)  project, 
which  aimed  to  "nowcast"  flu  prevalence  based  on  the  real-time  tabulation  of  query  entries.  In 
the  domain  of  global  public  health,  big  data  offers  the  possibility  not  only  of  facilitating  the 
epidemiological  tracking  of  disease,  but  of  forecasting  the  spread  of  disease  as  well,  thereby 
allowing  for  the  effective  and  timely  distribution  of  critical  resources  such  as  medication  and  aid 
workers.  Yet  projections  offered  by  the  GFT  wildly  overestimated  incidence  of  influenza  in  the 
US,  when  compared  against  doctors'  reports  collected  by  the  Center  for  Disease  Control  and 
Prevention.  Recent  analyses  of  this  failure  in  big  data  projection  have  revealed  systemic  problems 
in  the  use  of  such  data  sets  in  forecasting  models.12  Specifically,  overprojection  of  trend  models 
was  seen  to  result  from  failures  to  consider  the  uniqueness  of  individual  data  point.  In  the  case 
of  the  GFT,  a  cluster  of  regionalized  queries  probing  flu  symptomology  could  underlie  a  local 
outbreak  as  much  as  it  could  coincide  with  the  theme  of  a  district  school's  science  fair. 


12  Lazer,  D.,  Kenney,  R.,  King,  G.,  &  Vespignani,  A.  (2014).  The  parable  of  Google  Flu:  traps  in  Big  Data  analysis.  Science, 
343(6176):  1203-1205. 


23 


Misinterpretation  of  a  correlational  relationship  to  mean  a  causal  connection.  In  this  regard, 
the  interpretation  of  big  data  is  vulnerable  to  one  of  the  hallmark  errors  of  shoddy  statistics — 
the  misinterpretation  of  a  correlational  relationship  to  mean  a  causal  connection.  In  short,  when 
researchers  fail  to  consider  the  epistemological  heterogeneity  of  the  individual  data  points  within 
a  very  large  set— that  is,  the  meaning  underlying  the  behavioral  actions  which  have  been 
tabulated  and  accrued— their  failure  to  engage  with  the  ambiguity  inherent  in  the  data  set  may 
lead  to  a  dangerous  overfitting  of  their  predictive  models:  fundamental  misassumptions  about 
the  nature  of  the  data,  in  turn,  give  rise  to  inaccuracies  in  the  resulting  predictions. 

Reconciling  big  and  small  data.  One  of  the  central  tensions  revealed  in  this  pattern  is,  therefore, 
the  struggle  to  reconcile  big  and  small  data:  how  do  you  represent  the  specificity  of  individual 
points  within  the  big-picture  trends  revealed  by  large  and  complex  data?  Data  collected  through 
social  media  seems  to  promise  a  wealth  of  insight  on  broad  trends,  social  patterns,  and  consumer 
behaviors,  just  as  cell-phone  GPS  data  offers  unprecedented  tracking  of  commuting,  mobility, 
and  navigation  patterns  within  the  urban  environment.  And  yet  many  researchers  are  struggling 
with  the  problem  of  how  best  to  determine  data  validity  on  such  a  large  scale.  That  is,  how  do 
you  most  effectively  train  algorithms  to  distinguish  an  individual  user,  and  therefore  a  valid  data 
point,  from  unusable  data  generated  by  bots  and  advertisers?  How  do  you  tell  the  difference 
between  a  morning  commuter  and  an  out-of-town  tourist,  if  you  are  using  big  data  forecasts  to 
decide  where  to  construct  new  highways?  Do  these  distinctions  even  really  matter? 

Obscured  origins  of  epiphenomenon.  As  reliance  on  big  data  leads  to  the  decontextualization 
of  individual  data  points,  so,  too,  does  it  obscure  the  origins  of  epiphenomenon  arising  from  the 
nature  of  the  data  gathering  practice  itself.  For  instance,  in  the  weeks  leading  up  to  the  recent 
Scottish  independence  referendum  vote,  Amazon  DVD  charts  have  recorded  soaring  sales  of  the 
1995  epic  Braveheart,  which  vaulted  from  1074th  to  454th  place.13  And  yet  while  this 
phenomenon  was  short-lived,  big  data-driven  year-end  tabulations  of  DVD  sale  trends  now  run 
the  risk  of  over-representing  the  film's  general  popularity,  flattening  at  once  the  temporal 
transience  of  such  an  occurrence,  as  well  as  its  significance  as  a  social  artifact  of  a  particular 
historico-political  event. 

Big  data  offers  the  tantalizing  possibilities  of  gathering  unprecedented  quantities  of  rich 
information  in  real  time,  increasing  statistical  power  by  orders  of  magnitude  and  providing  new 
depth  and  perspective  to  our  understanding  of  the  operations  of  complex  socio-technical 
systems.  Ultimately  however,  these  few  case  examples  illustrate  that  accessing  big  data  alone  is 
not  enough  to  leverage  its  wide-ranging  potential;  rather,  the  ability  to  extract  meaningful 
forecasting  predictions  from  large  data  sets  relies  on  skillful  analytics  and  the  availability  of 
proper  tools  and  approaches  for  interactively  exploring  and  engaging  with  it  as  well.  In  this 
respect,  an  understanding  of  the  potentials,  and  pitfalls,  of  big  data  demands  a  rigorous 
consideration  of  both  the  capabilities  of  visual  analytic  and  the  nature  of  interactive  modeling 
approaches. 


13  Hooton,  Christopher.  "Scottish  Independence  Referendum  Leads  to  Surge  in  Sales  of  Braveheart."  The  Independent 
[London]  25  Sept.  2014. 
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Visual  Analytics 


Visual  analytics  is  resulting  in  a  transformative  capability,  bridging  human  and  computer  analysis. 

The  field  of  visual  analytics  has  grown  extensively  over  the  past  decade,  and  there  is  a  large  body 
of  knowledge  on  many  different  aspects.  In  Phase  1,  our  team  made  progress  in  finding  specific 
work  within  this  body  of  knowledge  of  specific  relevance  to  IMCSE,  and  this  will  continue  in  Phase 
2.  Much  of  the  work  on  visual  analytics  focuses  on  natural  data  and  fields  of  interest  outside  the 
scope  of  this  project  (e.g.,  biomedical,  marketing,  etc.).  Uncovering  the  most  salient  research 
findings  is  an  ongoing  effort. 

Uncertainty-Aware  Visual  Analytics.  Correa  et  al.  (2009)14  discuss  the  growth  of  visual  analytics 
as  an  important  tool  for  gaining  insights  on  large,  complex  data  sets.  The  authors  discuss  the 
problem  of  limitations  on  technology  and  human  power,  making  it  difficult  to  cope  with  the 
growing  scale  and  complexity  of  data,  and  therefore  making  it  is  seldom  possible  to  analyze  data 
in  its  raw  form.  The  data  must  be  transformed  to  a  suitable  representation  in  order  to  facilitate 
discovery  of  interesting  patterns.  However,  the  process  of  transforming  raw  data  to  abstractions 
and  derived  data  is  a  complex  network  of  transformations,  propagating  and  aggregating 
uncertainty.  As  such,  the  authors  believe  when  making  decisions  based  on  uncertain  data,  it  is 
important  to  quantify  and  present  to  the  analyst  both  the  aggregated  uncertainty  of  the  results 
and  the  impact  of  the  sources  of  that  uncertainty,  motivating  their  work  to  develop  a  framework 
for  uncertainty-aware  visual  analytics.  Figure  2  illustrates  the  process  developed  by  Correa  et  al., 
which  the  authors  describe  as  follows: 

In  general,  visual  analytics  is  the  process  of  transforming  input  data  into  insight.  A  similar 
process  occurs  for  the  uncertainty.  First,  uncertainty  modeling  generates  a  model  for 
source  uncertainty.  As  data  is  transformed,  these  uncertainties  are  propagated  and 
aggregated.  We  obtain  such  estimates  via  sensitivity  and  error  modeling.  Finally,  the 
uncertainty  on  the  derived  data  and  its  sources  are  mapped  to  visual  representations, 
which  finally  populate  the  view  used  by  the  analyst. 


14  Correa,  Carlos,  Chan,  Yu-Hsuan,  and  Ma,  Kwan-Liu,  "A  Framework  for  Uncertainty-Aware  Visual  Analytics",  IEEE 
Symposium  on  Visual  Analytics  Science  and  Technology,  2009,  p.  51-58 
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Figure  2.  Uncertainty-aware  visual  analytics  process  (source:  Correa,  et  ai.  200914) 


ANALYST 


Visual  Analytics  Based  Sensemaking.  Vitiello  and  Kalawsky  (2012)15  discuss  an  approach  that 
integrates  a  visual  analytic  based  workflow  to  the  notion  of  sensemaking.  The  authors  describe 
using  visual  analytics  to  support  systems  thinking  to  make  sense  of  complex  systems  interactions 
and  interrelationships  enabling  rapid  modeling  of  the  systems  of  interest  for  systems  engineering 
design  and  analysis  processes.  They  state  that  sensemaking  evolved  from  naturalistic  decision 
making  research,  as  published  by  Klein  et  al.  (1993)16.  The  visual  analytic  based  sensemaking 
framework  described  in  their  paper  is  aimed  toward  providing  the  means  to  rapidly  gain  valuable 
insights  into  the  data. 


Work-Centered  Approach  for  Visual  Analytics.  Yan  et  al.  (2012)  present  research  on  a  work- 
centered  approach  for  visual  analytics.  The  research  seeks  to  integrate  user-centered  design  and 
data-oriented  data-processing  algorithms  in  order  to  reconcile  human  users'  limited  capacity  to 
process  large  amount  and  rapid  growth  of  information  in  decision  making,  as  applied  to 
tradespace  exploration.  The  authors  state:  "After  a  user  selects  data  of  interest  from  raw  data, 
computational  algorithms  are  applied  to  build  data  models.  The  entire  model  building  process  is 
interactive  to  the  user.  User  has  the  capability  to  control  whether  and  how  algorithms  run  and 
constructs  a  specific  data  model  to  fit  ad-hoc  problems.  Visualization  provides  an  interface 
between  data,  models  and  the  user.  It  displays  both  source  data  and  computational  results.  It 
also  takes  user's  input  and  commands  to  manipulate  on  raw  data  or  analysis  algorithms". 17 


15  Vitiello,  P.  and  Kalawsky,  R.S.,  "Visual  analytics:  A  sensemaking  framework  for  systems  thinking  in  systems 
engineering",  IEEE  International  Systems  Conference  (SysCon),  2012 

16  Klein,  G.A.,  A  Recognition-Primed  Decision  (RPD)  Model  of  Rapid  Decision  Making,  Decision  making  in  action: 
Models  and  methods,  vol  5,  no  4,  pp  138-147,  Dec  1993. 

17  Yan,  X.,  Qiao,  M.,  Li,  J.,  Simpson,  T.W.,  Stump,  G.M.,  Zhang,  X.,  A  Work-Centered  Visual  Analytics  Model  to  Support 
Engineering  Design  with  Interactive  Visualization  and  Data-Mining,  Hawaii  International  Conference  on  Systems 
Science  (HICSS)  2012,  pl845-1854,  Jan  2012 
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Figure  3.  Framework  of  work-centered  visual  analytics  (source  Yan  et  al.17> 


Science  of  Interaction.  Our  inquiry  into  visual  analytics  necessitates  looking  into  the  "science  of 
interaction".  Pike  et  al.  discuss  the  interaction  challenges  raised  in  visual  analytics  research,  and 
the  relationship  between  interaction  and  cognition.18  The  'science  of  interaction1,  as  defined  by 
these  authors,  concerns  the  study  of  methods  by  which  humans  create  knowledge  through  the 
manipulation  of  an  interface.  They  state: 


/4s  visual  analytics  is  concerned  with  the  relationship  between  visual  displays  and  human 
cognition,  merely  developing  novel  visual  metaphors  is  rarely  sufficient  to  trigger  this 
insight  (where  insight  may  be  a  new  discovery  or  confirmation  or  negation  of  a  prior 
belief).  These  visual  displays  must  be  embedded  in  an  interactive  framework  that  scaffolds 
the  human  knowledge  construction  process  with  the  right  tools  and  methods  to  support 
the  accumulation  of  evidence  and  observations  into  theories  and  beliefs. 


Seven  key  areas  were  identified  in  this  2009  paper:  ubiquitous,  embodied  interaction;  capturing 
user  intentionality;  knowledge-based  interfaces;  principles  of  design  and  perception; 
collaboration;  interoperability;  and  interaction  evaluation.  Ongoing  research  in  these  areas  will 
be  explored  for  relevant  impact  as  our  research  project  progresses. 


Complex  Systems 

Developing  complex  systems  necessitates  an  approach  to  generate,  manage,  and  analyze 
artificial  data  across  all  aspects  of  system  complexity. 

The  growing  complexity  of  systems  is  well-recognized,  and  investigation  of  system  complexity  as 
related  to  engineered  systems  is  an  active  subject  of  inquiry.  Complexity,  for  instance,  can  relate 
to  the  number  of  constituent  and  component  interconnections,  and  to  the  necessary  rapid  rate 


18  Pike,  W.  A.,  Stasko,  J.,  Chang,  R.,  &  O'connell,  T.,A.  (2009).  The  science  of  interaction.  Information  Visualization, 
8(A),  263-274.  doi:http://dx.doi.org/10.1057/ivs.2009.22 
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of  information  generation  and  exchange.  It  can  also  relate  to  emergent  behavior  as  a  result  of 
interactions  of  constituent  systems  in  a  system  of  systems. 

Defining  Systems  Complexity.  Many  authors  have  and  continue  to  define  system  complexity. 
Gasper  (2012)19  discusses  three  bodies  of  work  than  can  be  used  as  a  basis  for  complexity 
definition  in  the  context  of  engineering.  Herbert  Simon  (1962)20  proposes  that  how  complex  or 
simple  a  structure  is  depends  critically  on  the  way  in  which  we  describe  it.  Simon  proposes  a 
hierarchical  approach  to  complexity,  decomposing  the  system  until  it  can  be  understood. 
Kolmogorov  (1983)21  definition  of  complexity  asserts  the  more  information  an  object  has,  the 
more  complex  it  is.  Given  the  system  is  the  object,  complexity  can  be  understood  as  related  to 
the  other  objects  that  interact  with  the  system.  The  specification  of  an  object  is  easier  when 
another  object  to  which  this  object  has  a  relation  is  already  specified.  A  third  work  by  Suh  (2005)22 
discusses  the  idea  of  information  connected  to  the  design  complexity,  proposing  that  the 
violation  of  the  information  axiom,  to  minimize  the  information  content  of  the  design  will 
maximize  the  probability  of  success,  will  result  in  complexity  in  the  system. 

Types  of  System  Complexity.  Structure  and  behavior  are  the  two  aspects  of  complex  systems 
addressed  in  classical  model-based  systems  engineering  23.  Rhodes  and  Ross  (2010)24  propose 
five  essential  aspects  for  the  engineering  of  complex  systems:  structural,  behavioral,  contextual, 
temporal,  and  perceptual.  They  argue  that  the  contextual,  temporal  and  perceptual  aspects  have 
been  under-addressed  in  engineering  methods,  and  have  past  and  ongoing  research  efforts  on 
advancing  the  constructs  and  methods  for  contextual,  temporal,  and  perceptual  aspects. 
Response  Systems  Comparison  is  a  resulting  method  to  address  the  five  aspects25.  The  method 
has  been  applied  in  various  domains  and  for  various  types  of  problems,  for  example.  Gasper19 
describes  the  application  for  a  conceptual  ship  design  problem. 


19  Gasper,  H.,  Rhodes,  D.,  Ross,  A.  and  Erikstad,  E.,  "Addressing  complexity  aspects  in  conceptual  ship  design:  a 
systems  engineering  approach",  Journal  of  Ship  Production  and  Design,  Vol.  28,  No.  4,  November  201 2,  pp.  1-15 

20  Simon,  H.,  "The  architecture  of  complexity".  Proceedings  of  the  American  Philosophical  Society,  106,  6,  467-482, 
1962 

21  Kolmogorov,  A.  N.,  "Combinatorial  foundations  of  information  theory  and  the  calculus  of  probabilities,  Russian 
Mathematical  Surveys,  38,  4,  27-36, 1983. 

22  Suh,  N.  P.,  "Complexity— theory  and  applications",  Oxford  University  Press,  Oxford,  UK,  2005 

23  Oliver,  D.,  Kelliher,  T.,  Keegan,  J.,  Engineering  Complex  Systems  with  Models  and  Objects,  NY:  McGraw-Hill,  1997. 

24  Rhodes,  D.H.  and  Ross,  A.M.,  "Five  aspects  of  engineering  complex  systems:  emerging  constructs  and  methods", 
Proceedings,  4th  Annual  IEEE  Systems  Conference,  April,  San  Diego,  CA,  2010 

25  Ross,  A.M.,  McManus,  H.L.,  Rhodes,  D.H.,  Hastings,  D.E.,  and  Long,  A.M.,  "Responsive  Systems  Comparison 
Method:  Dynamic  Insights  into  Designing  a  Satellite  Radar  System,"  AIAA  Space  2009,  Pasadena,  CA,  Sep  2009 


28 


STRUCTURAL 

related  to  the  form  of 
system  components  and 
their  inteireiationships 

“State  of  the  Practice”  systems 
architecting  and  design,  and 
emerging  model-based  systems 
engmeeiing  approaches 

BEHATTORAL 

related  to  peifonnance, 
operations,  and  reactions 
to  stimuli 

CONTEXTUAL 

related  to  circumstances 
in  which  the  system  exists 

New  constructs  and  methods  seek  to 
advance  "state  of  ail",  for  example: 

Epoch  Modeling 

Multi-Epoch  Analysis 

Epoch-Era  Analysis 
Multi-Stakeholder  Negotiations 
Visualization  of  Complex  Data  Sets 

TEMPORAL 

related  to  dimensions 
and  properties  of 
systems  over  time 

PERCEPTUAL 

related  to  stakeholder 
preferences,  perceptions 
and  cognitive  biases 

Figure  4.  Five  Aspects  of  Complex  Systems24 


Human-System  Interaction  Complexity.  The  complexity  of  the  human-system  interaction 
considerations  is  increasingly  important  in  developing  complex  systems.  A  2007  report  of  The 
National  Academies26  presents  a  discussion  of  the  challenges,  with  research  and  policy 
recommendations.  Many  of  the  points  brought  out  in  this  report  and  in  subsequent  work  extend 
to  understanding  of  complex  systems.  A  number  of  the  recommendations  have  extensions  to  the 
challenges  of  we  see  for  IMCSE,  and  are  beginning  to  be  addressed  through  research.  Examples 
include: 

•  Remote  collaboration  is  difficult  to  participate  in  or  observe  without  proper  remote 
collaboration  tools  enabling  interactivity  of  human  to  human,  and  human  to  model. 

•  Cognitive  and  perceptual  limitations  constrain  the  amount  of  information  that  can  be 
considered  at  a  point  in  time  by  a  single  decision  maker;  multi-sensory  representations 
may  allow  for  some  loosening  of  this  constraint  and  improve  human-model  interaction. 

•  Research  has  increasingly  uncovered  the  important  role  of  context  effects  on  both 
systems  in  use,  design,  and  on  the  decision  makers  themselves.  Facilities  that  can 
represent  and  control  for  these  context  effects  may  uncover  approaches  for  mitigating  or 
taking  advantage  of  these  effects. 

Perceived  and  Descriptive  Complexity.  Project  complexity  has  been  defined  in  many  different 
ways.  In  the  Applications  section  of  this  report,  we  hypothesize  that  perceived  and  descriptive 


26  National  Research  Council  (2007),  Human-Systems  Integration  in  the  System  Development  Process:  A  New  Look, 
Committee  on  Human-System  Design  Support  for  Changing  Technology,  R.W.  Pew  and  A.S.  Mavor,  Eds.,  Committee 
on  Human  Factors,  Division  of  Behavioral  and  Social  Sciences  and  Education,  Washington,  DC:  The  National 
Academies  Press. 
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complexity  are  correlated  and  constitute  a  tradeoff  between  design-efficiency  and  design- 
robustness  (refer  to  page  Error!  Bookmark  not  defined,  for  the  detailed  discussion). 


Model-Based  Systems  Engineering  (MBSE) 

Model-based  systems  engineering  generates  " artificial  data"  about  our  systems  which  we  use  to 
make  decisions  that  impact  the  future/continuing  success  of  that  system 

Systems  engineering  is  rapidly  becoming  model-based  in  nature.  It  is  recognized  that  MBSE  offers 
significant  potential27  but  many  challenges  remain  in  realizing  the  full  potential  of  using  models 
throughout  the  lifecycle  in  numerous  ways.  It  is  recognized  that  the  current  MPTs  are 
inadequate,  and  much  research  and  development  is  ongoing  to  address  this.  A  few  IMCSE- 
related  challenges  we  highlight  are:  integration  of  MPTs,  executable  artifacts,  issues  in  trusting 
models  and  need  for  ontologies. 

Inadequate  MPTs.  The  SERC's  System  2020  -  Strategic  Initiative  Report 28  states:  "Existing 
systems  engineering  tools,  processes,  and  technologies  poorly  support  rapid  design  changes  or 
capability  enhancements  within  acceptable  cost  and  schedule  constraints.  Their  focus  on  point 
solutions  makes  ad-hoc  adaptation  cumbersome  in  theatre.  To  increase  development  efficiency 
and  ensure  flexible  solutions  in  the  field,  systems  engineers  need  powerful,  agile,  interoperable, 
and  scalable  tools  and  techniques".  The  study  concluded  that  "the  purpose,  affordability,  and 
interoperability,  as  well  as  scalability  of  the  computer-aided  design  (CAD)  and  SE  tools  available 
to  DoD  were  weak  with  respect  to  the  complexities  of  future  DoD  missions  and  net-centric 
systems  of  systems."  These  findings  underscore  the  motivation  for  evolving  model-based 
systems  engineering  MPTs  to  enable  users  to  interact  with  models  in  a  more  effective  manner. 

Executable  system  architecture  artifacts.  According  to  the  recent  study  by  the  Systems 
Engineering  Division  of  NDIA29,  "Model  Based  Engineering  (MBE)  is  an  emerging  approach  to 
engineering  that  holds  great  promise  for  addressing  the  increasing  complexity  of  systems,  and 
systems  of  systems,  while  reducing  the  time,  cost,  and  risk  to  develop,  deliver,  and  evolve  these 
systems".  The  study  assessed  the  current  state  of  MBE  and  identified  potential  benefits,  costs 
and  risks  within  the  DoD  acquisition  lifecycle  context. 

According  to  a  recent  SERC  study:30 


27  Zimmerman,  P.,  A  Review  of  Model-Based  Systems  Engineering  Practices  and  Recommendations  for  Future 
Directions  in  the  Department  of  Defense,  2nd  Systems  Engineering  in  the  Washington  Metropolitan  Area  (SEDC  2014) 
Conference,  Chantilly,  VA,  April  3,  2014 

28  SERC-2010-TR-009-1,  Boehm,  B.,  System  2020  -  Strategic  Initiative,  Systems  Engineering  Research  Center,  Final 
Technical  Report,  August  26,  2010. 

29  NDIA  Systems  Engineering  Division,  Modeling  and  Simulation  Committee,  Final  Report  of  the  Model  Based 
Engineering  (MBE)  Subcommittee,  Feb  10,  2011. 

30  SERC-2012-TR-024,  zur  Muehlen,M.,  Integration  of  M&S  (Modeling  and  Simulation,  Software  Design  and  DoDAF, 
RT  24,  Systems  Engineering  Research  Center  (SERC),  April  9,  2012 
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“ Modeling  and  Simulation  (M&S)  technology  are  essential  to  understand  the  behavior  of 
the  target  system  and/or  to  evaluate  various  strategies  for  the  operation  of  the  system 
before  it  is  actually  built.  In  many  cases,  simulation  models  reflect  the  design  of  the  final 
system  in  great  detail  and  can  take  the  place  of  architecture  documentation.  In  an  ideal 
scenario,  system  architecture  artifacts  should  be  directly  executable  and  could  be 
leveraged  for  simulation  purposes  ”. 

Creating  requisite  process  and  data  models,  as  well  as  use  case  descriptions  can  facilitate  the 
transition  from  requirements  engineering  to  simulation,  and  to  implementation.  The  transition 
from  design  to  implementation,  however,  is  not  seamless.  Differences  in  tool-specific  standard 
implementations  hamper  the  seamless  transition  of  model  information,  increasing  the  burden 
on  the  user. 

Trust  in  Constructed  Models.  A  recent  SERC  study  sponsored  by  the  Naval  Air  Systems  Command 
(NAVAIR),  Introducing  Model-Based  Systems  Engineering:  Transforming  System  Engineering 
through  Model-Based  Systems  Engineering 31  assessed  the  technical  feasibility  of  creating  and 
leveraging  a  more  holistic  MBSE  approach.  The  vision  for  "doing  everything  with  models" 
depends  on  a  common  lexicon  for  MBSE  including  model  levels,  types,  uses,  and  representations, 
and  a  significant  degree  of  automation.  The  sophisticated  model-based  process  and  enabling 
environment  that  are  envisioned  offer  the  potential  for  a  very  powerful  transformation  of 
systems  engineering  through  MBSE.  A  very  significant  challenge  in  realizing  such  a  vision  is  trust 
in  constructed  models36,  as  we  discuss  below. 

Ontology  for  Human  Systems  Integration.  A  recent  publication  by  Orellana  and  Madni  (2014)32 
discusses  the  importance  of  creating  the  ontology  for  human  systems  interaction,  interfaces,  and 
integration.  An  ontology,  according  these  authors,  will  "extend  current  system  modeling 
capabilities  that  will  enable  the  human  element  to  be  analyzed  as  part  of  the  overall  system 
development  process.  As  posed  in  this  paper,  the  role  of  the  human  as  system  operator  is 
evolving  to  that  of  agent,  placing  greater  demands  on  system  architects  and  engineers33.  The 
ontology,  when  developed,  will  "extend  current  modeling  capabilities  and  allow  the  human 
element  to  be  analyzed  as  part  of  the  overall  system  from  system  conception  to  system 
disposal32. 


Emerging  Challenges  at  the  Intersection 

Across  the  four  topics  areas,  we've  identified  several  emerging  challenges  at  their  intersection 
with  regard  to  IMCSE.  The  insights  and  techniques  being  developed  in  each  of  the  four  topic 
areas  have  particular  additional  considerations  when  used  to  support  systems  engineering  and 


31  SERC-2014-TR-044,  Blackburn,  M.,  Transforming  Systems  Engineering  through  Model  Based  Systems  Engineering, 
Technical  Report,  Systems  Engineering  Research  Center  (SERC),  March  31,  2014 

32  Orellana,  D.  and  Madni,  A.,  Human  System  Integration  Ontology:  Enhancing  Model  Based  Systems  Engineering  to 
Evaluate  Human-System  Performance,  Conference  on  Systems  Engineering  Research,  2014,  Procedia  Computer 
Science  28  (2014)  19-25 

33  Madni,  A.,  Integrating  Humans  with  software  and  Systems,  Technical  Challenges  and  a  Research  Agenda,  Systems 
Engineering  2010,  13  (3),  232-245 
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decision  making.  Each  of  these  challenges  will  now  be  briefly  described.  We  anticipate  using 
these  challenges  to  help  orient  and  motivate  some  of  the  research  activities  within  IMCSE. 

Tradeoff  of  Models 


Central  to  most  analyses  are  models.  Since  every  model  is  an  abstraction  from  reality,  it  is 
important  for  any  model  user  to  understand  the  implications  of  embedded  assumptions. 
Sensitivity  analysis  is  a  step  often  performed  during  analyses  where  the  stability  of  results  is 
investigated,  as  a  function  of  (often  parametric)  assumptions  (Feuchter  2000)34.  "Sensitivity 
analyses  should  be  performed  whenever  time  and  resources  allow,  with  an  emphasis  on 
alternatives  that  survived  early  screening  processes"  (OAS  2008)35.  In  practice,  many  studies  are 
resource  constrained  and  therefore  only  cursory  (if  any)  sensitivity  analysis  is  conducted.  Since 
the  assumptions  in  the  models  impact  the  results  of  those  models,  not  only  are  choices  of  model 
parameters  important  from  a  "within"  model  sensitivity  perspective,  but  also  the  choice  of  the 
model  itself  can  have  large  ramifications  on  the  results.  IMCSE  will  seek  to  address  the  challenge 
of  performing  broad  sensitivity  analysis,  in  terms  of  model  choice,  as  part  of  a  given  study,  so 
that  it  is  not  relegated  to  a  later  activity  that  is  subject  to  omission  when  resources  are  short. 

Some  preliminary  research  was  done  to  trade  "within  model"  sensitivities  in  value  models, 
investigating  the  potential  for  interaction  in  refining  value  model  parameter  choices  (Ricci  et  al. 
2014). 36 
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Figure  5.  Trust  and  truthfulness  in  value  models  (Ricci  et  al.  2014) 


IMCSE  will  continue  to  develop  techniques  and  frameworks  for  conducting  trades  on  models 
themselves  and  not  just  within  the  data  generated  by  the  models.  One  example  exploratory 
project  is  described  in  the  "Value  Model  Tradeoff"  section  later  in  this  report. 


34  Feuchter,  C.A.,  "Air  Force  Analyst's  Handbook:  On  Understanding  the  Nature  of  Analysis,"  Office  of  Aerospace 
Studies,  Air  Force  Materiel  Command  (AFMC)  OAS/DR,  Kirtland  AFB,  NM,  www.oas.kirtland.af.mil,  January  2000. 

35  Office  of  Aerospace  Studies  (OAS),  "Analysis  of  Alternatives  (AoA)  Handbook:  A  Practical  Guide  to  Analyses  of 
Alternatives,"  Air  Force  Materiel  Command  (AFMC)  OAS/A9,  Kirtland  AFB,  NM,  www.oas.kirtland.af.mil,  July  2008. 

36  Ricci,  N.,  Schaffner,  M.A.,  Ross,  A.M.,  Rhodes,  D.H.,  and  Fitzgerald,  M.E.,  "Exploring  Stakeholder  Value  Models  Via 
Interactive  Visualization,"  12th  Conference  on  Systems  Engineering  Research,  Redondo  Beach,  CA,  March  2014 
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Visual  Analytics  of  Artificial  (Model-generated)  Data 


Much  of  the  visual  analytics  literature  highlights  particular  computational  and  user  interaction 
techniques,  or  supporting  infrastructure,  or  applications  to  particular  cases.  Validation  of 
proposed  techniques  and  supporting  infrastructure  typically  hinges  on  matching  user-generated 
insights  and  predictions  to  "ground  truth"  in  the  data.  This  means  that  the  particular  data  set 
being  explored  via  VA  tends  to  be  rooted  in  natural  (i.e.  empirical)  data,  where  "ground  truth" 
has  meaning.  Once  this  is  the  case,  visualizations  for  pattern  matching  by  humans  is  more  likely 
to  be  uncovering  actual  patterns  in  the  data,  rather  than  artifacts.  (Artifacts  may  still  exist  due 
to  data  errors,  sensor  errors,  or  data  abstraction  and  aggregation  effects,  for  example.  However, 
these  effects  can  be  managed  if  a  valid  (i.e.  "true")  dataset  is  available.)  An  example,  of  this 
dynamic  is  displayed  in  the  MIT  Big  Data  Challenge.  In  this  contest,  a  large  data  set  of  historical 
taxi  data  and  other  related  data  sets  are  provided  to  competitors.  Competitors  must  develop 
predictive  models  of  number  of  taxi  trips  as  a  function  of  location.  The  scoring  of  the  predictive 
models  "will  be  computed  as  the  root-mean-squared  error  of  [the]  predictions  against  the  ground 
truth."37 

Since  the  goal  of  visual  analytics  is  to  generate  insights  into  relationships  and  patterns  in  the  data, 
the  existence  of  potentially  confounding  artifacts  in  the  data  makes  it  especially  challenging  when 
ground  truth  is  no  longer  available.  This  is  essentially  the  difference  between  exploratory 
modeling  and  consolidative  modeling  (Bankes  1993)38.  Consolidative  modelling  includes 
"techniques  in  which  known  facts  are  consolidated  into  a  single  model"  in  order  to  generate 
explanatory  relationships  of  existing  data  (Kwakkel  and  Pruyt  2012). 39  While  in  exploratory 
modelling,  the  intention  is  to  "generate  artificial  data"  that  "can  inform  modelers  and  decision 
makers  of  the  ramifications  of  various  sets  of  assumptions,  as  well  as  provide  consistent 
communication"  (Schaffner  2014)40. 

In  IMCSE,  models  will  tend  to  be  of  exploratory  nature  and  therefore  additional  considerations 
must  be  taken  into  account  when  generating  and  visualizing  the  data  in  order  to  properly 
interpret  the  results. 

Perceptual  and  Cognitive  Considerations  in  Human-Model  Interaction 

In  considering  the  form  of  visual  analytics  to  represent  big  data,  and  the  structure  of  model-based 
approaches  to  forecasting  the  evolving  complexities  of  large-scale  system,  it  is  crucial  to  also 
consider  the  perceptual  and  cognitive  capabilities  of  human  beings  at  the  center  of  these 
exploratory  efforts. 


37  http://bigdatachallenge.csail.mit.edu/prediction  [accessed  9/29/2014] 

38  Bankes,  S.  (1993),  "Exploratory  Modeling  for  Policy  Analysis,"  Operations  Research,  4,  pp:  435-449, 
doi:10.1287/opre.41.3.435. 

39  Kwakkel,  J.H.,  and  Pruyt,  E.,  "Exploratory  Modelling  and  Analysis,  an  approach  for  model-based  foresight  under 
deep  uncertainty  ",  Technol.  Forecast.  Soc.  Change  (2012),  http://dx.doi.Org/10.1016/i.techfore.2012.10.005  . 

40  Schaffner,  M.A.,  Designing  Systems  for  Many  Possible  Futures:  The  RSC-based  Method  for  Affordable  Concept 
Selection  (RMACS),  with  Multi-Era  Analysis,  Master  of  Science  Thesis,  Aeronautics  and  Astronautics,  MIT,  June  2014. 
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There  are  many  considerations  in  human-model  interaction,  and  relevant  research  crosses 
multiple  disciplines.  For  example,  recent  neurocognitive  investigations  offer  some  insight  into 
three  behavioral  phenomena  related  to  decision-making  which  may  provide  a  structural 
framework  for  guiding  these  considerations:  1)  the  behavioral  over-reliance  on  cognitive  biases 
in  choice  behavior;  2)  the  tendency  towards  ambiguity  aversion;  and  3)  the  limitations  of 
affective  forecasting  when  making  projections  about  future  needs  and  desires. 

Behavioral  Over-Reliance  on  Cognitive  Biases  in  Choice  Behavior.  Broadly  speaking,  cognitive 
biases  arise  from  a  maladaptive  overreliance  on  heuristics,  a  series  of  cognitive  'short-cuts' 
human  beings  recruit  to  reduce  the  complexity  of  day-to-day  decisions,  and  thereby  decrease 
cumulative  cognitive  loading.41  Heuristics  allow  individuals  to  extrapolate  from  the  consequences 
of  previous  decision-making  events  to  inform  future  choice  behavior.  Yet  when  individuals 
become  overly  reliant  on  such  strategies,  at  the  exclusion  of  considering  novel,  situation-specific 
information,  they  become  biased,  and  often  demonstrate  impaired  decision-making  abilities. 
However,  research  has  suggested  that  cycles  of  cognitive  bias  can  be  broken  through  training  and 
self-monitoring,  and  that  the  effective  visual  presentation  of  information  may  reduce  a  reliance 
on  biased  strategies  and  promote  thoughtful  consideration  of  salient  data  points,  leading  in  turn 
to  better  and  more  informed  choice  patterns. 

Human  Tendency  towards  Ambiguity  Aversion.  A  second  important  neurocognitive 
consideration  is  the  processing  of  information  regarding  risk  and  ambiguity.  Recent  studies 
investigating  the  neural  correlates  of  decision-making  have  shown  distinct  patterns  of  brain 
activation  in  response  to  uncertainty.42'43  Behaviorally,  it  has  been  well  established  that  a  risky 
option  of  known  probability— even  when  the  odds  are  poor— is  often  favored  over  one  where 
the  decider  is  ignorant  of  the  precise  degree  of  risk,  a  phenomenon  termed  'ambiguity 
aversion.'  44  Broadly  speaking,  these  findings  point  to  a  general  human  intolerance  for  ambiguity 
and  a  preference  for  information  seeking.  In  this  regard,  one  of  the  advantages  of  big  data  driven, 
model-based  forecasts  is  to  reduce  ambiguity  by  extrapolating  future  patterns  from  previously 
observed  occurrences,  thereby  generating  new  and  useful  information  for  decision-makers. 

Limitations  of  Affective  Forecasting  When  Making  Projections.  Finally,  efforts  in  experimental 
psychology  to  probe  human  abilities  to  'affectively  forecast'— that  is,  to  make  accurate 
projections  about  their  future  wants,  desires,  and  emotional  states— have  revealed  that,  on 
average,  people  are  in  fact  quite  inaccurate  in  determining  the  emotional  consequences  of  future 
events,  often  overestimating  the  amount  of  future  satisfaction  a  given  set  of  events  will  bring 


41  Tversky,  A.,  &  Kahneman,  D.  (1974).  Judgement  under  uncertainty:  Heuristics  and  biases.  Sciences ,  185(4157): 
1124-1131. 

42  Brand,  M.,  Labudda,  K.,  &  Markowitsch,  H.  J.  (2006).  Neuropsychological  correlates  of  decisionmaking  in 
ambiguous  and  risky  situations.  Neural  Networks,  19(8),  1266-1276. 

43  Huettel,  S.  A.,  Stowe,  C.  J.,  Gordon,  E.  M.,  Warner,  B.  T.,  &  Platt,  M.  L.  (2006).  Neural  Signatures  of  Economic 
Preferences  for  Risk  and  Ambiguity.  Neuron,  49(5),  765-775. 

44  Fox,  C.,  Tversky,  A.,  1995.  Ambiguity  aversion  and  comparative  ignorance.  The  quarterly  Journal  of  Economics, 
110(3),  585-603. 
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them.45  To  this  end,  model-based  forecasts  which  project  future  states  by  examining  current  and 
past  trends  may  prove  essential  to  aiding  decision-making  about  the  future  which  may  otherwise 
be  tainted  by  inaccurate  assumptions  of  impending  affective  state. 

When  taken  together,  these  three  streams  of  neurocognitive  research  highlight  the  ways  in  which 
a  fundamental  understanding  of  people's  perceptual  and  cognitive  capabilities  allows  for  rich, 
human-centered  design  in  the  presentation  of  visual  analytic  displays  and  engaging,  interactive 
models  which  facilitate  exploration  and  discovery.  At  the  same  time,  work  from  these  fields  also 
illuminates  ways  in  which  visual  analytic  approaches  and  model-based  projections  may  serve  as 
effective  aids  for  complex,  real-world  decision-making. 


45  Wilson,  T.D.;  Gilbert,  D.T.  (2003).  "Affective  Forecasting".  Advances  in  Experimental  Social  Psychology,  35:  345- 
411. 
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Fundamentals 


The  Fundamentals  thrust  presently  includes  three  areas:  Interactive  Epoch-Era  Analysis  project, 
Value-Model  Choice  and  Tradeoff,  and  Supporting  MPTs. 


Interactive  Epoch-Era  Analysis 

Epoch-Era  Analysis  is  a  framework  that  supports  narrative  and  computational  scenario  planning 
and  analysis  for  both  short  run  and  long  run  futures.  This  project  is  performing  exploratory 
development  of  interactive  Epoch-Era  Analysis,  including  human  interface  and  reasoning 
considerations  for  epoch  and  era  characterizations,  as  well  as  single  and  multi-  epoch/era 
analyses. 


Background 

Epoch-Era  Analysis  (EEA)  is  an  approach  designed  to  clarify  the  effects  of  changing  contexts  over 
time  on  the  perceived  value  of  a  system  in  a  structured  way  (Ross  200646,  Ross  and  Rhodes 
200847).  The  base  unit  of  time  in  EEA  is  the  epoch,  which  is  defined  as  a  time  period  of  fixed 
needs  and  context  in  which  the  system  exists.  Epochs  are  represented  using  a  set  of  epoch 
variables,  which  can  be  continuous  or  discrete  values.  These  variables  can  be  used  to  represent 
any  exogenous  uncertainty  that  might  have  an  effect  on  the  usage  and  perceived  value  of  the 
system;  weather  conditions,  political  scenarios,  financial  situations,  operational  plans,  and  the 
availability  of  other  technologies  are  all  potential  epoch  variables.  Appropriate  epoch  variables 
for  an  analysis  include  key  (i.e.,  impactful)  exogenous  uncertainty  factors  that  will  affect  the 
perceived  success  of  the  system.  A  large  set  of  epochs,  differentiated  using  different  enumerated 
levels  of  these  variables,  can  then  be  assembled  into  eras,  ordered  sequences  of  epochs  creating 
a  description  of  a  potential  progression  of  contexts  and  needs  over  time.  This  approach  provides 
an  intuitive  basis  upon  which  to  perform  analysis  of  value  delivery  over  time  for  systems  under 
the  effects  of  changing  circumstances  and  operating  conditions,  an  important  step  to  take  when 
evaluating  large-scale  engineering  systems  with  long  lifecycles. 

Encapsulating  potential  short  run  uncertainty  (i.e.  what  epoch  will  my  system  experience  next?) 
and  long  run  uncertainty  (i.e.  what  potential  sequences  of  epochs  will  my  system  experience  in 
the  future?)  allows  analysts  and  decision  makers  to  develop  dynamic  strategies  that  can  enable 
resilient  systems.  Key  challenges  in  application  of  EEA  up  to  this  point  involve  eliciting  a 
potentially  large  number  of  relevant  epochs  and  eras,  conducting  analysis  across  these  epochs 
and  eras,  and  extracting  useful  and  actionable  information  from  the  analyses.  Schaffner  (2014)48 


46  Ross,  A.M.,  "Managing  Unarticulated  Value:  Changeability  in  Multi-Attribute  Tradespace  Exploration,"  MIT 
Engineering  Systems  Division  PhD  thesis,  2006. 

47  Ross,  A.M.  and  D.H.  Rhodes,  "Using  Natural  Value-Centric  Time  Scales  for  Conceptualizing  System  Timelines 
through  Epoch-Era  Analysis,"  INCOSE  2008,  2008. 

48  Schaffner,  M.A.,  Designing  Systems  for  Many  Possible  Futures:  The  RSC-based  Method  for  Affordable  Concept 
Selection  (RMACS),  with  Multi-Era  Analysis,  Master  of  Science  Thesis,  Aeronautics  and  Astronautics,  MIT,  June  2014. 
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showed  that  the  number  of  potential  eras  to  consider  grows  very  quickly,  becoming 
computationally  infeasible. 

As  an  example,  a  system  model  represented  by  5  epoch  variables,  each  with  3  levels,  would  result 
in  35  =  243  possible  epochs.  If  the  length  of  our  eras  is  10  epochs  and  each  epoch  can  transition 
between  any  other  epoch,  then  the  size  of  the  potential  era  space  would  be  24310  ~1024.  This 
means  that  for  many  problem  formulations  it  is  not  feasible  to  evaluate  systems  across  all  or 
even  a  large  fraction  of  potential  eras49.  Research  in  both  the  areas  of  big  data  analysis  and  visual 
analytics  have  led  to  techniques  that  could  be  leveraged  to  mitigate  these  challenges.  It  is 
hypothesized  in  this  research  that  augmenting  the  traditional  EEA  approach  with  new  analytic 
and  interactive  techniques  will  fundamentally  enable  new  capabilities  and  insights  to  be  derived 
from  EEA,  resulting  in  superior  dynamic  strategies  for  resilient  systems.  In  particular,  we  have 
three  informal  hypotheses  regarding  interactive  EEA  (IEEA): 

1.  IEEA  will  enable  the  elicitation  of  more  broad/complete  set  of  possible  epochs. 

a.  Infrastructure  that  enables  IEEA  could  include  databases  of  epoch  variables,  which 
could  be  leveraged  in  future  IEEA  studies. 

b.  Explicit  implementations  in  an  interface  will  provide  repeatable  and  more 
understandable  elicitation  experiences,  resulting  in  more  epoch  variables. 

2.  IEEA,  through  a  human-in-the-loop  implementation,  will  help  to  intelligently  limit  the 
potentially  unbounded  growth  in  the  epoch/era  space. 

a.  Using  visual  analytic  techniques  such  as  filtering,  binning,  pattern  matching, 
search  algorithms,  and  human-in-the-loop  interaction,  IEEA  can  be  used  to 
effectively  manage  multi-epoch  and  multi-era  analysis  scale  growth. 

3.  IEEA  will  enable  the  development  of  superior  intuition,  buy-in,  and  insight  generation  for 
decision-making. 

a.  By  allowing  decision  makers  to  "experience"  (i.e.  "see"  and  "interact  with") 
epochs  and  eras,  they  will  better  understand  and  accept  the  impact  of  context  and 
needs  changes  on  systems  and  therefore  how  resilience  can  be  better  achieved. 

Earlier  work  demonstrated  promise  for  such  capability  and  insight  improvement  when 
interactivity  is  added  to  tradespace  exploration.  Ross  et  al.  (2010)50  introduces  a  method,  applied 
to  two  aerospace  cases  in  order  to  explore  the  potential  for  interactive  tradespace  exploration 
to  support  stakeholder  negotiations.  Preliminary  results  indicate  the  method  to  be  a  rapid  and 
beneficial  technique,  which  generated  compromise  alternatives,  guided  the  elicitation  of 
previously  unarticulated  information,  and  resulted  in  increased  confidence  and  solution  buy-in 
of  participating  stakeholders.  Interactive  tradespace  exploration  analyses  allowed  negotiation 
processes  to  proceed  quickly.  Proposed  compromises  can  be  assessed  by  each  stakeholder  in 
real  time,  and  what  the  stakeholder  is  gaining  or  losing  in  the  compromise  is  immediately  visible. 


49  Schaffner  2014  suggested  several  possible  mitigations  to  this  problem,  including  human  in  the  loop  era  tree 
pruning,  which  will  be  investigated  in  this  research  project. 

50  Ross,  A.M.,  McManus,  H.L.,  Rhodes,  D.H.,  and  Hastings,  D.E.,  "A  Role  for  Interactive  Tradespace  Exploration  in 
Multi-Stakeholder  Negotiations,"  AIAA  Space  2010,  Anaheim,  CA,  September  2010. 
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An  open  area  of  research  is  to  incorporate  Epoch-Era  Analysis  into  the  interactive  tradespace 
exploration. 


Introduction 

The  development  of  Engineered  Resilient  Systems  (ERS)  was  identified  as  a  science  and 
technology  (S&T)  priority  for  the  DoD  by  the  Secretary  of  Defense  in  April  2011.  Since  that  time 
several  researchers  and  practitioners  have  begun  to  develop  methods,  techniques  and  tools  to 
assist  designers  in  the  early  system  concept  selection  phase.  Many  of  the  techniques  in 
development  require  analysis  of  large  amounts  of  data  to  quantify  the  effectiveness  of  large 
numbers  of  actionable  alternatives  across  large  numbers  of  possible  futures  in  order  to  support 
the  best  possible  decision.  To  assist  in  solving  the  stated  problem,  this  research  will  leverage  and 
expand  upon  some  human-in  the-loop  techniques  that  are  emerging  in  studies  of  visual  analytics 
and  big  data  analysis.  The  challenge  this  research  seeks  to  address  can  be  described  as:  "how 
can  one  balance  System,  Context,  and  Expectations  over  time,  during  engineering  design, 
evaluation  and  selection,  given  human  cognitive  and  perceptual  limitations?"  (Ross  2014)51 

The  development  of  complex  engineering  systems  using  traditional  engineering  design 
techniques  can  lead  to  point  designs  optimized  for  a  fixed  operating  context  or  set  of  stakeholder 
needs.  This  can  reduce  system  performance  if  future  uncertainty  resolves  in  a  way  other  than 
predicted.  This  is  especially  true  if  the  system  is  not  resilient  or  robust  to  change.  As  an  example, 
consider  modern  spacecraft,  which  have  long  development  timelines  of  5  to  10  years  or  more 
that  makes  them  susceptible  to  changes  in  mission  and  technology  before  they  even  reach  orbit. 
They  must  also  have  a  significant  amount  of  redundancy  built  in  because  a  replacement  system 
could  take  years  to  develop  and  launch  if  they  fail.  Reducing  such  susceptibilities  to  changes  in 
context  was  a  key  goal  of  DARPA's  System  F6  program.  A  shift  in  stakeholder  needs  for  which 
the  system  is  not  resilient  can  also  limit  its  value  delivery.  A  noteworthy  example  is  the  Iridium 
satellite  constellation  that  suffered  from  a  shift  in  the  consumer  market  to  land-based  cellular 
towers  before  it  reached  initial  operating  capability  (IOC)  (Curry  2014)52. 

The  definition  of  what  is  or  is  not  a  resilient  system  is  not  universally  agreed  upon,  and  how  it  has 
been  defined  and  measured  in  past  studies  has  varied  across  problem  domains  (Goerger  et  al 
2014)53.  One  definition  is  that  a  resilient  system  has  "the  ability  to  circumvent,  survive,  and 
recover  from  failures  to  ultimately  achieve  mission  priorities  even  in  the  presence  of 
environmental  uncertainty"  (Madni  2012)54.  Yet  another  definition  of  resilience  (called  system 
"survivability"  elsewhere,  adding  to  semantic  confusion)  is  "the  ability  of  a  system  to  minimize 


51  Ross,  A.M.,  "Interactive  Model-Centric  Systems  Engineering,"  Presentation  to  the  5th  Annual  SERC  Sponsor 
Research  Review,  Georgetown  University,  Washington,  DC,  February  2014. 

52  Curry,  M.,  "Presentation:  Application  of  Epoch  Era  Analysis  to  the  Design  of  Engineered  Resilient  System",  17th 
NDIA  Systems  Engineering  Conference,  Springfield,  VA,  October,  2014. 

53  Goerger  S,  Madni  A,  Eslinger  0.,  "Engineered  Resilient  Systems:  A  DoD  Perspective,"  12th  Conference  on  Systems 
Engineering  Research,  Redondo  Beach,  CA,  March  2014. 

54  Madni,  A.,  "Affordable,  Adaptable  and  Effective:  The  Case  for  Engineered  Resilient  Systems,"  Engineering  Resilient 
Systems  Workshop,  Pasadena,  CA,  August  2012. 
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the  impact  of  a  finite  duration  disturbance  on  value  delivery,  achieved  through  either  (1)  the 
reduction  of  the  likelihood  or  magnitude  of  a  disturbance;  (2)  the  satisfaction  of  a  minimally 
acceptable  level  of  value  delivery  during  and  after  a  finite  disturbance  or;  (3)  timely  recovery 
from  a  disturbance  event"  (Richards  et  al.  2007)55. 

What  is  common  to  most  of  the  definitions  suggested  for  resilient  systems  is  an 
acknowledgement  that  complex  systems  must  be  designed  to  continue  to  deliver  sustained  value 
to  their  stakeholders  even  if  uncertainty  exists  about  the  way  a  system  will  be  required  to  operate 
in  the  future.  More  recent  work  has  generalized  this  concept  into  something  called  value 
sustainment  (Beesemyer  20  1  256).  Value  sustainment  is  defined  as  "the  ability  to  maintain  value 
delivery  in  spite  of  epoch  shifts  or  disturbances."  Figure  6  below  summarizes  this  concept  and 
reflects  how  we  will  consider  notions  of  resilience  in  this  research  effort.  In  this  figure,  the 
nominal  value  delivered  by  a  system  is  (potentially)  impacted  by  a  perturbation  (characterized  as 
either  a  disturbance  or  a  shift).  A  disturbance  is  a  short  duration,  likely  to  revert  imposed  change 
on  the  design,  context,  or  needs  for  a  system,  while  a  shift  is  a  long  duration,  unlikely  to  revert 
imposed  change  on  the  design,  context,  or  needs  for  a  system.  A  "resilient"  system  is  one  that 
either  is  not  impacted,  or  maintains  value  above  the  indicated  threshold,  and  restores  that  value 
delivery  to  a  higher  acceptable  level  after  a  threshold  period  of  time. 

a.  "Long"  duration  perturbation  b.  "Short"  duration  perturbation 

(unlikely  to  revert)  (likely  to  revert) 


Figure  6.  Long  (a)  and  short  (b)  run  impacts  of  perturbations  on  value  delivery 


T RADITIONAL  EEA  AND  DATA  CHALLENGES 

Traditional  tradespace  exploration  and  multidisciplinary  design  optimization  techniques  typically 
assume  as  fixed  the  needs  of  the  stakeholders,  the  context  in  which  a  system  will  be  operated 
and  the  future  state  of  the  system  itself.  To  design  resilient  systems  we  must  consider  situations 
in  which  these  can  all  vary  with  time.  One  framework  for  evaluating  such  possibilities  is  Epoch 


55  Richards,  M.G.,  Hastings,  D.E.,  Rhodes,  D.H.,  and  Weigel,  A.L.,  "Defining  Survivability  for  Engineering  Systems,"  5th 
Conference  on  Systems  Engineering  Research,  Hoboken,  NJ,  March  2007. 

56  Beesemyer,  J.C.,  Empirically  Characterizing  Evolvability  and  Changeability  in  Engineering  Systems,  Master  of 
Science  Thesis,  Aeronautics  and  Astronautics,  MIT,  June  2012. 
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Era  Analysis  (Ross  2006)57.  EEA  conceptualizes  the  effects  of  time  and  changing  context  on  a 
system  by  modeling  combinations  of  future  context  and  stakeholder  needs  on  perceived  system 
value  (Ross  and  Rhodes  200858;  Fitzgerald  et  al.  201159;  Schaffner  et  al.  201360).  A  time  period 
over  which  the  stakeholder  needs  and  the  context  in  which  the  system  must  operate  are  fixed  is 
referred  to  as  an  epoch.  A  series  of  epochs  can  be  strung  together  to  form  eras  that  can  be  used 
to  model  the  long-run  value  delivery  of  a  system  and  take  into  account  temporal  path 
dependencies  between  epochs.  Such  eras  can  be  generated  through  narrative  (i.e.  story-driven) 
or  computational  means  (i.e.  algorithm-generated)  enabling  consideration  of  a  broader  set  of 
possible  short  and  long  run  scenarios  than  commonly  considered  using  traditional  scenario 
planning  techniques  (Roberts  et  al.  2009)61. 


Figure  7.  Activities  in  Epoch-Era  Analysis 


Broadly  speaking,  EEA  be  described  as  the  following  activities  (roughly  sequential  and  depicted 

in  Figure  7): 

0.  Problem  Definition:  identify  decision  to  be  made,  relevant  constraints,  stakeholders,  and 
potential  contexts 

1.  Design  Formulation:  generate  potential  design  alternatives  to  be  evaluated  in  the  analysis;  can 

be  generated  via  inheritance,  creative  brainstorming,  value-driven  methods,  or  other 
means;  identify  preliminary  criteria  for  their  evaluation. 

2.  Epoch/Era  Generation: 

a.  Epoch  Characterization:  identify  key  exogenous  uncertainties  and  parameterize 
via  epoch  variables;  can  be  accomplished  via  era  deconstruction  or  proposing 
possible  short  run  scenarios. 

b.  Era  Construction:  generate  various  long  term  descriptions  of  possible  futures  via 
epoch  sequencing,  or  proposing  long  run  scenarios  (e.g.  via  narrative  or 
computational  means). 

3.  Design-Epoch-Era  Evaluations:  develop  and  execute  appropriate  models  that  can  evaluate 

designs  in  epochs  and  eras. 


57  Ross,  A.M.,  Managing  Unarticulated  Value:  Changeability  in  Multi-Attribute  Tradespace  Exploration,  Doctor  of 
Philosophy  Dissertation,  Engineering  Systems  Division,  MIT,  June  2006. 

58  Ross,  A.M.,  and  Rhodes,  D.H.,  "Using  Natural  Value-centric  Time  Scales  for  Conceptualizing  System  Timelines 
through  Epoch-Era  Analysis,"  INCOSE  International  Symposium  2008,  Utrecht,  the  Netherlands,  June  2008. 

59  Fitzgerald,  M.E.,  Ross,  A.M.,  and  Rhodes,  D.H.,  "A  Method  Using  Epoch-Era  Analysis  to  Identify  Valuable 
Changeability  in  System  Design,"  9th  Conference  on  Systems  Engineering  Research,  Los  Angeles,  CA,  April  2011. 

60  Schaffner,  M.A.,  Wu,  M.S.,  Ross,  A.M.,  and  Rhodes,  D.H.,  "Enabling  Design  for  Affordability:  An  Epoch-Era  Analysis 
Approach,"  Proceedings  of  the  10th  Annual  Acquisition  Research  Symposium-  Acquisition  Management,  April  2013. 

61  Roberts,  C.J.,  Richards,  M.G.,  Ross,  A.M.,  Rhodes,  D.H.,  and  Hastings,  D.E.,  "Scenario  Planning  in  Dynamic  Multi- 
Attribute  Tradespace  Exploration,"  3rd  Annual  IEEE  Systems  Conference,  Vancouver,  Canada,  March  2009. 
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4.  Single  Epoch/Era  Analysis: 

a.  Single  Epoch  Analysis:  conduct  analyses  of  the  designs  within  particular  epochs, 
determining  performance  and  cost  of  alternatives  and  difficulty  of  achieving 
success  within  particular  periods  of  fixed  context  and  needs. 

b.  Single  Era  Analysis:  conduct  analyses  within  particular  eras  to  determine  the 
impact  of  time-dependent  effects  on  system  success,  along  with  cumulative  path- 
dependence  on  the  system  over  time. 

5.  Multi  Epoch/Era  Analysis: 

a.  Multi-Epoch  Analysis:  conduct  analysis  across  multiple  (or  all)  epochs  to 
determine  sensitivities  of  designs  to  epochs;  gives  insight  into  short  run  value  of 
active  and  passive  strategies  for  system  resilience. 

b.  Multi-Era  Analysis:  conduct  analysis  across  multiple  (or  all)  eras  to  determine 
sensitivities  of  designs  to  eras  and  patterns  of  path  dependence;  gives  insights  into 
long  run  value  of  active  and  passive  strategies  for  system  resilience. 


Era 


v 


Frame  0 


Frame  1 


Frame  2 


Frame  K 


Need  1 

Need  2 

Need  N 

Need  1 

Need  2 

Need  N 

Context  1 

Context  1 

Context  1 

Context  2 

Context  2 

Context  M 

Possible  epochs  in  each  frame 

Figure  8.  Era-tree  showing  potential  temporal  paths  through  the  epoch  space  (based  on  Ross  et  al.  200862) 

Figure  8  illustrates  the  era-tree  approach  to  era  construction  via  paths  through  the  epoch  space. 
Each  epoch  is  defined  as  a  particular  context-need  pair  and  duration.  A  frame  is  a  particular  slot 
within  an  era  that  consists  of  an  epoch  and  a  duration  (Schaffner  2014)63.  This  allows  an  EEA  user 
to  specify  eras  of  varying  number  of  slots  in  a  less  ambiguous  manner.  For  example,  a  5  frame 
era  consists  of  5  slots,  each  with  a  particular  epoch  and  duration.  The  same  epoch  could  appear 
in  more  than  one  frame.  A  second  useful  concept  is  that  of  a  clip,  which  is  a  subset  of  a  full  era, 
comprised  of  an  arbitrarily  small  number  of  frames.  Using  this  nomenclature,  one  can  speak  of 


62  Ross,  A.M.,  McManus,  H.L.,  Long,  A.,  Richards,  M.G.,  Rhodes,  D. H .,  and  blastings,  D.E.,  "Responsive  Systems 
Comparison  Method:  Case  Study  in  Assessing  Future  Designs  in  the  Presence  of  Change,"  AIAA  Space  2008,  San 
Diego,  CA,  September  2008 

63  Schaffner,  MA,  Designing  Systems  For  Many  Possible  Futures:  The  RSC-Based  Method  For  Affordable  Concept 
Selection  (RMACS),  With  Multi-Era  Analysis,  SM  in  Aeronautics  and  Astronautics,  Cambridge,  MA:  MIT,  2014. 
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3-frame  clips,  for  example,  which  might  appear  in  multiple  different  eras.  When  looking  for 
patterns,  such  a  unit  of  analysis  may  be  useful. 


a.  b.  Epoch 

Figure  9.  Epochs  as  Alternative  "Point"  Futures  (a)  and  Multi-Epoch  Analysis  (b) 

Figure  9  illustrates  epochs  as  alternative  (point)  futures,  and  multi-epoch  analysis  as  a  cross¬ 
epoch  activity  looking  for  designs  that  perform  well  across  the  alternative  future  space. 

As  previously  noted,  a  practical  challenge  in  implementations  of  EEA  is  the  large  amount  of  data 
that  may  need  to  be  evaluated  in  order  to  thoroughly  characterize  possible  system  alternatives 
and  their  potential  for  value  sustainment  across  a  wide  variety  of  futures.  Notably,  trends  in  the 
area  of  ERS-related  research  are  moving  towards  analysis  of  tradespaces  on  the  order  of  multiple 
terabytes  of  data.  Drawing  on  recent  research  in  the  areas  of  big  data  analysis  and  visual 
analytics,  EEA  methods  can  be  augmented  to  allow  a  decision  maker  to  interactively  filter,  sort, 
aggregate,  and  identify  patterns  in  the  data  more  efficiently  than  predetermined  or  automated 
algorithms,  enabling  a  more  effective  tradeoff  of  evaluation  "completeness"  versus  insights 
gained. 

Liu  et  al.  (2013)64  and  Heer  and  Shneiderman  (2012)65  point  out  that  "interaction  is  essential  to 
exploratory  visual  analysis",  but  their  work  primarily  focuses  on  visualization.  Note  that 
interaction,  as  used  here,  is  not  intended  to  be  strictly  limited  to  the  data  visualization 
component,  but  also  the  interfaces,  processes,  and  methods  that  allow  a  user  to  gain  insights 
from  their  data.  Interfaces  may  require  use  of  sensory  stimuli  other  than  visual-only,  including 
touch  and/or  sound.  Processes  could  also  include  custom  workflows  such  as  those  described  in 
Sitterle  et  al.  (2014)66.  Methods  for  sorting  and  filtering  data  may  include,  but  are  not  limited  to, 
interactive  brushing  and  linking  of  multiple  coordinated  visual  displays. 

The  problems  that  may  arise  when  scaling  up  to  larger  decision  problems  with  traditional  EEA 
can  be  placed  into  four  categories: 

1.  Data  size  increases  which  creates  a  storage  and  data  transmission  problem. 

2.  Data  size  increase  also  creates  a  separate  problem  related  to  cross-filtering  across  large 
numbers  of  data  dimensions.  Human  cognitive  limitations  make  comprehension  of  high- 


64  Liu,  Z.  Jiang,  B.,  Heer,  J.,  "imMens:  Real-time  Visual  Querying  of  Big  Data,"  Eurographics  Conference  on 
Visualization  (EuroVis),  2013. 

65  Heer,  J.,  and  Shneiderman,  B.,  "Interactive  Dynamics  for  Visual  Analysis,"  ACM  Queue,  2012. 

66  Sitterle,  V.,  Curry,  M.,  Ender,  T.,  Freeman,  D.,  "Integrated  Toolset  and  Workflow  for  Tradespace  Analytics  in 
Systems,"  INCOSE  International  Symposium,  Las  Vegas,  NV,  2014. 


42 


dimensional  data  difficult  so  datasets  must  be  "sliced"  or  cross  tabulated  across 
dimensions  before  rendering  them  as  ID,  2D  or  3D  visualizations. 

3.  Larger  data  sets  require  increased  amounts  of  processing  time  to  manipulate. 

4.  Rendering  problems  arise  when  large  amounts  of  data  must  be  visualized  simultaneously. 

Solutions  to  these  and  other  issues  relevant  to  IEEA  will  be  discussed  in  the  following  sections. 
Demonstration  cases  that  test  applicability  of  various  research  methods  will  also  be  discussed. 


Enabling  Areas  of  Research 

Several  areas  of  research  that  will  enable  IEEA  and  address  or  mitigate  the  issues  previously 
discussed  have  been  identified.  The  sections  below  describe  background  research  on  various 
techniques  and  ongoing  efforts  to  extend  them  for  IEEA  applications. 

Data  Reduction  Methods 


Problems  with  rendering  and  the  scalability  of  visualizations  and  other  encoded  visual 
information  can  be  improved  upon  using  techniques  that  do  not  require  every  single  data  point 
to  be  drawn.  Liu  points  out  that,  "Perceptual  and  interactive  scalability  should  be  limited  by  the 
chosen  resolution  of  the  visualized  data,  not  the  number  of  records,"  and  summarizes  several 
techniques  past  researchers  have  applied  to  reduce  the  pixel  density  of  visualizations  including 
(1)  filtering;  (2)  sampling;  (3)  binned  aggregation;  and  (4)  model-fitting  (Liu  et  al.  2013). 67 

Filtering  is  a  commonly  applied  technique  to  reduce  the  problem  to  a  subset  of  the  original  data. 
This  is  accomplished  by  placing  bounds  on  the  data  such  that  not  all  of  it  is  displayed  at  once 
which  could  be  overwhelming  to  a  decision  maker.  Likewise,  sampling  is  a  technique  for  reducing 
the  amount  of  data  displayed  to  the  user  by  randomly  drawing  a  subset  of  the  points  to  create  a 
reduced  set  for  display.  Sampling  has  the  potential  downside  of  unintentionally  concealing 
features  of  the  dataset  that  may  correspond  to  rare  events.  These  are  oftentimes  the  very  data 
points  in  which  a  decision  maker  is  most  interested. 

Filtering  and  sampling  are  often  used  in  practice  because  they  are  relatively  easy  to  implement 
and  do  not  require  any  changes  in  the  standard  visualizations  types  that  would  be  used  for  a 
larger  dataset.  Both  techniques  are  useful,  but  we  would  also  like  to  consider  techniques  that 
allow  all  the  data  to  be  visualized.  Binned  aggregation  is  powerful  in  that  it  allows  a  decision 
maker  to  observe  global  patterns  in  the  data  as  well  as  local  features  that  may  be  hidden  by 
filtering  or  sampling  (Liu  et  al.  20  1  3). 68  In  TSE  and  EEA,  which  often  use  2-D  scatter  plots  to 
display  data,  one  example  of  binned  aggregation  is  to  project  the  data  into  a  1-D  histogram. 
Alternatively,  the  data  could  also  be  aggregated  into  smaller  2-D  bins  with  the  density  of  points 


67  Liu,  Z.  Jiang,  B.,  Heer,  J.,  "imMens:  Real-time  Visual  Querying  of  Big  Data,"  Eurographics  Conference  on 
Visualization  (EuroVis),  2013. 

68  Liu,  Z.  Jiang,  B.,  Heer,  J.,  "imMens:  Real-time  Visual  Querying  of  Big  Data,"  Eurographics  Conference  on 
Visualization  (EuroVis),  2013. 
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encoded  by  color.  Examples  of  these  techniques  are  demonstrated  in  prototype  visualization 
tools  described  in  the  following  sections. 

Model-fitting  is  another  approach  that  can  be  applied  to  reduce  the  resources  required  to 
visualize  a  large  dataset.  Examples  of  model-fitting  include  simple  regression  models  or  complex 
surrogate  models  that  reduce  the  dataset  to  representative  equations.  Model-fitting  can  be  a 
powerful  technique,  but  computing  an  appropriate  model  can  sometimes  be  computationally 
expensive.  Models  also  typically  have  some  amount  of  error  in  how  well  they  represent  the 
underlying  data  and  this  must  be  carefully  considered  when  using  their  outputs  for  the  purpose 
of  decision-making. 

Online  Analytical  Processing 

Online  Analytical  Processing  (OLAP)  is  an  approach  for  creating  abstract  representations  of  high¬ 
dimensional  datasets.  OLAP  is  frequently  applied  in  data  mining  and  other  exploratory  analysis 
applications  with  large  amounts  of  data.  These  datasets  are  often  stored  in  relational  databases 
with  multiple  tables  connected  by  keys,  but  can  also  be  as  simple  as  a  spreadsheet  with  records 
stored  in  each  row  and  with  columns  representing  different  attributes  or  properties  of  the  data. 
In  fact,  pivot  tables  generated  in  MS  Excel  are  one  example  of  a  common  application  of  OLAP  for 
summarizing  data.  A  notable  application  of  OLAP  is  its  successful  use  in  business  intelligence 
applications  to  parse  large  amounts  of  sales,  cost  and  other  data  to  evaluate  trends  and  inform 
business  decisions. 

For  IEEA,  the  benefit  of  OLAP  is  that  it  enables  a  user  to  view  data  from  multiple  points  of  view 
and  quickly  uncover  previously  undiscovered  relationships  and  patterns  within  the  dataset.  A 
decision-maker  looking  at  a  large  number  of  candidate  designs  across  a  large  possible  epoch 
space  can  apply  OLAP  techniques  to  slice,  dice,  drill  down,  roll  up  or  compute  pivots  of  the  hyper¬ 
dimensional  data  cube  representing  design  alternatives  over  epochs  and  eras.  This  allows  them 
to  easily  extract  data  that  is  of  interest  to  them  which,  in  turn,  enables  better  intuition  on  which 
to  base  decisions. 

Human  Interaction  Methods 


As  a  component  of  the  IEEA  research,  several  concepts  related  to  how  humans  interact  with  their 
data  are  being  examined.  Interaction  methods  may  extend  beyond  visualization  approaches  to 
include  touch  and  auditory  interaction  as  well.  For  the  effort  presented  here,  however,  we  will 
primarily  focus  on  research  related  to  visual  techniques.  In  this  paper,  we  use  multiple 
coordinated  views  and  animated  transitions  as  approaches  for  facilitating  deeper  understanding 
of  the  data.  Multiple  coordinated  views  can  be  used  in  exploratory  visualization  to  more 
effectively  expose  relationships  in  the  underlying  data.  Coordinated  views  are  separate, 
independent  views  of  a  given  set  of  data  that  serve  as  complementary  representations,  and  may 
aid  in  identifying  patterns  as  well  as  errors  in  the  data.  The  individual  views  of  the  data  are  not 
intended  for  use  in  isolation,  but  rather  to  be  combined  to  generate  insights.  The  primary 
purpose  of  coordinated  visualizations  is  to  allow  improved  understanding  through  user 
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interaction  with  different  simultaneous  representations  of  the  data  (Roberts  2007)69.  While 
choosing  which  combinations  of  views  to  use  in  order  to  generate  insights  can  be  complicated, 
several  guidelines,  including  compactness  and  diversity  of  the  visualizations,  have  been  discussed 
in  prior  literature  (Scherr  2009)7°. 

Search  Algorithms 

As  the  number  of  design  and  epoch  variables  increase,  TSE  and  EEA  techniques  quickly  become 
computationally  expensive  due  to  the  non-linearity  between  number  of  variables  and  number  of 
model  evaluations  required  (Schaffner  2014)71.  This  can  be  true  even  if  model  evaluations  can 
be  computed  relatively  quickly.  In  his  research,  Schaffner  explores  application  of  both  breadth- 
first  and  depth-first  algorithms  to  improve  the  computational  efficiency  of  multi-era  analysis. 
This  area,  however,  remains  an  important  area  of  further  research  to  enable  IEEA. 


69  Roberts  J.,  "State  Of  The  Art:  Coordinated  &  Multiple  Views  In  Exploratory  Visualization,"  5th  Int'l  Conf  on 
Coordinated  and  Multiple  Views  in  Exploratory  Vis.  Washington,  DC,  2007. 

70  Scherr  M.,  "Multiple  And  Coordinated  Views  In  Information  Visualization,"  Media  Informatics  Advanced  Seminar 
on  Info  Vis.  2008/2009 

71  Schaffner  MA.,  Designing  Systems  For  Many  Possible  Futures:  The  RSC-Based  Method  For  Affordable  Concept 
Selection  (RMACS),  With  Multi-Era  Analysis,  SM  in  Aeronautics  and  Astronautics,  Cambridge,  MA:  MIT,  2014. 
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A  Framework  for  Interactive  Epoch-Era  Analysis 


The  current  vision  of  Interactive  Epoch-Era  Analysis  (IEEA)  leverages  humans-in-the-loop 
interaction,  as  well  as  supporting  infrastructure,  in  order  to  manage  challengers  associated  with 
the  large  amounts  of  data  potentially  generated  in  a  study,  as  well  as  to  improve  sense  making 
of  the  results.  Figure  10  below  illustrates  three  insertion  points  for  interactivity  to  directly 
address  the  three  hypotheses  outlined  earlier  (i.e.,  improved  elicitation,  improved  analyses,  and 
improved  decision-making). 

As  shown  in  the  figure,  many  of  the  techniques  discussed  above  can  be  applied  to  augment  the 
existing  EEA  workflow.  OLAP  techniques  may  be  applied  to  advance  current  data  handling,  and 
search  algorithms  may  improve  our  ability  to  offer  more  informed  recommendations  to  decision¬ 
makers  during  the  epoch-era  elicitation  process.  Similarly,  enhanced  human  interaction 
techniques  and  visualizations  may  aid  in  the  analyses  of  the  vast  amounts  of  information  required 
to  reach  an  informed  decision. 


Figure  10.  Interactive  Epoch-Era  Analysis  leverages  humans-in-the-loop  analysis  and  supporting  infrastructure 


When  considering  human-interaction  in  EEA  (in  both  current  and  potential  implementations), 
one  can  see  a  large  number  of  potential  tasks  that  can  be  addressed.  These  include,  but  are  not 
limited  to,  the  following:  elicitation  of  the  objectives  to  be  met  (e.g.  evaluation  criteria)  as  well 
as  categories  of  uncertainties  (e.g.  epoch/era  categories),  as  well  as  design  alternatives  to 
consider;  generation  of  epoch  factors  within  the  uncertainty  categories,  along  with  the  epoch 
variables  used  to  quantitatively  represent  the  epoch  factors,  as  well  as  the  ranges  and  allowable 
enumerations  for  each  epoch  variable;  sampling  of  which  particular  epochs  and  eras  to 
eventually  evaluate;  development  and  execution  of  evaluations  of  designs  in  epochs  in  eras; 
analyses  of  designs  within  particular  epochs  and/or  eras  as  well  as  across  many  particular  epochs 
and/or  eras;  and  finally  decision-making  through  synthesis  of  gathered  evidence,  perhaps 
through  iterative  refinement. 
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A  key  challenge  in  EEA  is  that  the  number  of  possible  epochs  generated  by  enumerating  epoch 
variables  can  quickly  exceed  a  feasible  number  for  users  to  explore,  potentially  resulting  in  biased 
or  uninformative  analysis.  For  IEEA  to  be  effective,  it  must  enable  effective  management  of  this 
challenge.  The  proposed  IEEA  framework,  in  which  certain  EEA  activities  are  performed  as  a 
partnership  between  computer  and  human  feedback,  seeks  to  enable  more  informative  and 
satisfactory  selections  and  analyses.  The  above  framework  has  been  extended  in  Figure  11,  in 
order  to  make  explicit  particular  workflow  considerations  (i.e.,  the  above  tasks)  for  human-in- 
the-loop  IEEA. 


Figure  11.  A  framework  for  Interactive  Epoch-Era  Analysis,  showing  five  "modules"  with  human  interaction 


This  framework  can  be  abstracted  into  six  main  modules: 

Elicitation  of  relevant  epoch  and  design  variables  (often  through  interview), 
Generation  of  all  epochs  and  design  tradespaces  (often  including  enumeration), 
Sampling  of  epochs  and  eras  in  which  to  evaluate  design  choices, 

Evaluation  of  designs  in  sampled  subset  of  epochs  and  eras 

Analyses  of  design  choices  in  the  previously  evaluated  epochs  and  eras,  and  finally 

Decisions  of  final  designs  based  on  iterative  evidence  from  previous  modules. 


While  the  sequence  of  these  modules  flows  logically,  IEEA  is  intended  to  be  an  iterative  process 
where  users  can  go  back  and  change  responses  within  earlier  modules  at  any  point  to  reflect 
what  they  have  learned  from  later  ones. 
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In  the  past  elicitation  and  generation  have  been  primarily  a  human-centric  task,  with  some 
structured  support  via  static  documentation;  sampling,  however,  is  the  first  module  in  the 
framework  that  can  clearly  benefit  from  human-computer  interaction  and  feedback.  In  this 
module,  the  human  must  make  sense  of,  and  decide  upon,  which  subset  of  epochs  and  eras  to 
spend  computational  and  human  attention  (i.e.,  scarce)  resources.  Visualization  and  feedback 
are  key  tasks  for  the  user  in  order  to  interact  with  the  data  representing  possible  epoch  and  era 
subset  samples  from  the  generated  larger  epoch  and  era  spaces.  For  now,  we  focus  on  this 
module  as  a  proof  of  concept  before  expanding  considerations  to  other  modules.  In  the  following 
subsections  we  present  and  evaluate  different  visualization  options  and  considerations  for  use  in 
the  epoch/era  sampling  module,  and  then  propose  metrics  for  usability  analysis,  with  the  idea 
that  this  kind  of  analysis  can  (and  should)  be  conducted  on  other  interactivity  modules  as  well  in 
future  implementations  of  IEEA. 
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Visualization  Considerations  for  Multi-Dimensional  Data 


Background  on  "Good"  Visualizations 

Before  examining  techniques  that  could  be  used  in  epoch  sampling  visualization,  we  set  a 
baseline  for  good  overall  design  principles.  There  are  countless  guidelines  that  have  been 
developed  over  the  years  that  prescribe  measures  to  be  taken  when  creating  data  visualizations. 
These  range  from  very  general  (e.g.  "Tell  the  truth  about  the  data,"  "Important  data  should  be 
easy  to  find  and  understand")  to  very  specific  (e.g.  "Colors  should  be  chosen  so  that  all,  including 
color-blind,  users  can  distinguish  them,"  "Avoid  using  gray  scale  to  represent  more  than  2-4 
values,"  "Words  should  be  spelled  out  and  run  horizontally,  left-to-right").  There  should  be 
internal  consistency  within  the  visualizations,  as  well  as  external  consistencies  with  common 
conventions  the  user  may  be  familiar  with.  Visualizations  should  attract  the  viewer  to  think  about 
the  substance  rather  than  the  methodology  or  any  other  distracting  features.  They  should  be 
clear  and  reveal  the  data  at  several  levels  of  detail,  attracting  and  encouragingthe  userto  explore 
further.  Above  all,  they  should  enable  the  user  to  be  more  productive,  efficient,  and/or  gain  more 
insight  than  they  could  have  without  the  tool  (Ware  2013;  Tufte  1983)72'73. 

Especially  when  dealing  with  quantitative  data,  it  is  important  to  take  into  account  how  different 
values  are  encoded  to  reflect  their  size  or  order.  According  to  a  1984  study  by  Cleveland  and 
McGill,  viewers  are  most  accurately  able  to  encode  quantitative  data  in  the  following  ways,  in 
order  (Cleveland  1984) 74: 

1)  Position  along  a  common  scale  (e.g.  scatter  plots) 

2)  Position  along  nonaligned  scales  (e.g.  multiple  scatter  plots) 

3)  Length,  direction,  angle/slope  (e.g.  bar  chart,  pie  chart) 

4)  Area  (e.g.  bubbles) 

5)  Volume,  curvature  (e.g.  spheres) 

6)  Shading,  color  saturation  (e.g.  heatmap) 

Thus  representations  of  quantitative  (or  even  some  categorical)  information  should  take  this  list 
into  account. 

While  the  more  general  guidelines  are  more  obvious  and  widely  accepted,  more  specific  ones 
should  be  treated  with  caution,  as  not  all  users  share  the  same  visual  preferences.  Regardless,  all 
of  these  guidelines  strive  to  optimize  the  processing  power  of  human  cognitive  ability  to 
understand  whatever  is  being  displayed.  Tufte  summarizes  the  concept  of  graphical  excellence 
as  "that  which  gives  to  the  viewer  the  greatest  number  of  ideas  in  the  shortest  time  with  the 


72  Ware,  Colin.  Information  Visualization:  Perception  for  Design.  Elsevier,  2013. 

73  Tufte,  Edward  R.  The  Visual  Display  of  Quantitative  Information.  Cheshire,  Conn.:  Graphics,  1983. 

74  Cleveland,  W.S.  and  R.  McGill,  "Graphical  Perception:  Theory,  Experimentation,  and  Application  to  the 
Development  of  Graphical  Methods,"  Journal  of  the  American  Statistical  Association,  79-387, 1984. 
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least  ink  in  the  smallest  space."  For  all  of  the  types  of  visualizations  we  are  about  to  introduce, 
we  credit  that  these  baseline  design  criteria  are  heeded  and  satisfied. 


Considerations  for  Sampling  Module  in  IEEA 

We  now  turn  to  the  sampling  module  of  IEEA  to  examine  the  effectiveness  of  different 
visualization  types.  Sampling  epochs,  as  mentioned  in  the  previous  section,  is  made  difficult  by 
the  rate  at  which  the  number  of  possible  epochs  grows.  While  computers  are  good  at  handling 
vast  amounts  of  data,  it  takes  human  judgment  to  decide  which  epochs  are  most  likely, 
important,  or  urgent  to  analyze  further.  Thus,  human-in-the-loop  interaction  is  necessary  to 
achieve  effective  epoch  sampling  results.  We  recall  the  main  goals  for  this  module,  adapted  from 
Curry  et  al.'s  hypotheses  regarding  IEEA  (Curry  2015)75: 

1)  The  user  should  understand  how  each  of  the  epochs  are  defined  in  the  dataset  (e.g. 
epoch  variables  and  values;  what  is  a  context  and  what  is  a  need,  etc). 

2)  Based  on  this,  the  user  should  be  able  to  find  and  select  important  epochs  on  which  to 
conduct  further  analysis. 

3)  Finally,  the  user  should  understand  a)  the  size  of  the  epoch  space,  b)  what  fraction  is 
available  to  explore  (for  which  epochs  data  has  already  been  generated),  c)  what  fraction 
of  this  has  already  been  explored  or  selected  to  explore,  helping  to  "intelligently  limit  the 
potentially  unbounded  growth  in  the  epoch/era  space." 

In  summary,  these  are  the  goals  (and  implied  evaluation  criteria)  for  potential  visualizations: 

Goal  #1:  help  user  understand  specific  epoch  definitions 

Goal  #2:  help  user  find  and  select  epoch(s) 

Goal  #3:  help  user  understand  a)  epoch  space  size,  b)  fraction  available  to  explore,  c) 

fraction  already  explored 

In  the  next  section  we  will  describe  an  existing  implementation  and  evaluate  it  on  these  three 
goals.  In  following  sections  we  will  introduce  five  selected  visualization  techniques,  describing 
and  evaluating  the  strengths  and  weaknesses  of  each,  both  for  IEEA  based  on  the  three  goals 
above,  and  for  general  use. 


Existing  Implementation  in  IVTea  Suite 

A  sampling  module  is  currently  implemented  as  the  first  step  in  the  Interactive  Value-Driven 
Tradespace  Exploration  and  Analysis  Suite  (IVTea).  IVTea  Suite  begins  with  the  user  loading  a 
dataset,  which  contains  a  pre-elicited  and  generated  dataset.  Users  are  then  required  to  specify 
an  epoch  subset,  or  construct  an  era,  using  a  simple  drop-down  menu  that  contains  a  name  or  ID 
number  of  all  of  the  possible  epochs  in  the  dataset  (see  Figure  12).  Other  "widgets"  within  IVTea 
suite  do  allow  users  to  gain  more  information  about  the  definition  of  the  epochs  (via  ID  numbers. 


75  Curry,  M.D.  and  A.M.  Ross,  "Considerations  for  an  Extended  Framework  for  Interactive  Epoch-Era  Analysis,"  CSER 
2015. 
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and  context  variables  and  preference  sets),  however  these  are  not  readily  apparent  to  a  novice 
user.  The  interface  does  work  (albeit  a  little  inefficiently)  for  experienced  users,  who  understand 
the  epochspace,  labels,  and  where  to  find  important/useful  epochs,  but  it  provides  minimal 
guidance  to  a  new  user  in  completing  the  main  goals. 


Welcome 


Epoch  Knobs 


File  Options  Help 
No  epochs  «v«M>le. 


Select  epoch" 

Drop  downs 
„„  &  lists 


Welcome  to  fVTeel  Where  wotid  you  Ike  to  begn? 

Des-cn  Era 


*  EeowaiedVaUetOnV 


\ 

Preference  Sets 


«i.2.coer}) 

;«1.3.CO*f}} 

((M/coaT)} 


_ Potential  epoch _ 

a  space  fraction  Epoch  Fiitef 
F,l,  Options  H't/  Custom 

VaW Epochs  n-«(50W%)  FUteTS 

Add  Fave  1  ^ 

tf«#r  by  Techkevei . v*«  •  SO  00%) 


Add  fter  on^TPChocse  a  vane  We 


TechCevel  _ 

Vi  000  to  2  0001  Le**  Than 


JiL 


Select  epoch 


Figure  12.  IVTea  Suite  example  interfaces  for  finding  and  selection  task 

Keeping  in  mind  a  user  may  not  be  the  same  one  that  generated  the  information  in  the  first  place, 
the  interface  should  make  the  tasks  of  understanding  and  finding/selection  readily  apparent. 
Less  experienced  users,  or  those  unfamiliar  with  the  particular  dataset,  will  not  know  how  to 
interpret  poorly  labeled  epochs  (most  likely  failing  Goal  #1),  much  less  which  are  important  for 
analysis,  and  will  most  likely  choose  epochs  near  the  top  of  the  drop-down,  biasing  their  eventual 
analysis  results  (most  likely  failing  Goal  #2).  The  initial  interface  ("Epoch  Knobs")  does  not  allow 
for  examining  the  properties  of  an  epoch  beyond  what  is  shown  in  the  drop-down,  and  the  only 
epochs  displayed  are  the  ones  the  user  is  able  to  explore,  leaving  no  sense  of  the  total  epoch 
space  or  the  fraction  already  explored  (failing  Goal  #3).  Users  must  use  a  combination  of  "Epoch 
Knobs"  and  "Epoch  Filter"  for  the  task  of  epoch  finding  and  selection  (Goal  #2)  (via  specification 
of  an  epoch  as  a  "favorite").  For  the  task  of  epoch  understanding  (Goals  #1  and  #3),  the  user 
must  use  a  combination  of  "Summary  Dash"  to  drill  down  to  "Epoch  Summary"  and  "Context 
Summary"  and  "Preference  Summary"  (see  Figure  13).  We  hypothesize  adding  more  effective 
targeted  visualization,  along  with  appropriate  interactivity  would  improve  the  learning  process 
for  new  users. 
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Figure  13.  IVTea  Suite  interface  examples  for  the  understanding  epochs  and  epoch  spaces  tasks 


Next  we  examine  a  selection  of  possible  visualization  techniques  made  capable  by  D3.js,  a 
JavaScript  library  for  enabling  interactive  data-driven  web  documents76. 


Possible  Visualization  Implementations 

This  section  will  consider  a  number  of  alternative  visualization  techniques  with  a  description, 
example  implementation,  and  evaluation  in  terms  of  supporting  the  three  goals  outlined  above. 


Scatter  Plots  and  Bubble  Charts 
Description 

As  shown  from  Cleveland  and  McGill's  aforementioned  study,  people  are  most  accurately  able  to 
decode  information  when  it  is  represented  by  position  along  a  common  scale,  making  a  scatter 
plot  a  good  place  to  start.  A  scatter  plot  allows  one  variable  to  be  plotted  on  each  axis,  so  each 
point's  location  easily  encodes  its  characteristics  to  new  users.  A  bubble  chart  is  a  scatter  plot 
with  two  additional  values  encoded  using  color  and  size.  Figure  14  shows  an  example  of  a  scatter 
plot,  while  Figure  15  shows  an  example  of  a  bubble  chart. 


76  Data-Driven  Documents  (www.d3js.org) 
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Figure  14.  Example  of  scatter  plot  from  [77].  X-variable  is  Education,  Y-variable  is  Income 


Figure  15.  Example  of  bubble  chart  from  [7S1.  X-variable  is  Cost,  Y-variable  is  Profits,  Color  variable  is  Project 

name,  and  Size  variable  is  Probability  of  Success 


77  http://www.texample.net/media/tikz/examples/PNG/scatterplot.png 

78  http://www.bubblechartpro.com/content/images/Bubble_Chart_Example_3.jpg 
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Example  implementation  and  evaluation  of  goals 


Scatter  plots  are  virtually  the  best  way  to  represent  two-dimensional  data,  so  if  there  are  only 
two  epoch  variables,  a  scatter  plot  may  be  the  best  way  to  visually  represent  the  entire  epoch 
space.  A  rudimentary  implementation  of  an  interface  using  a  scatter  plot  using  16  epochs  is 
shown  in  Figure  16. 


X-axis:  1  Tech  Level  j  ) ,  Y-axis:  User  Preference  $ 
Selected  Epochs:  16  Reset  Selections 


8n 


6  - 


2 


Mure  present 

Figure  16.  Example  of  IEEA  Epoch  Sampling  implemented  as  a  scatter  plot.  The  epoch  variables  were  "Tech 
Level,"  with  values  "future"  or  "present,"  and  "User  Preference,"  with  values  1-8. 

Given  only  two  epoch  variables,  scatter  plots  clearly  show  the  combination  of  the  epoch  variables 
defining  each  epoch  (Goal  #1),  and  based  on  this,  a  user  can  easily  locate  and  select  epochs  of 
interest  (Goal  #2).  All  possible  epoch  alternatives  can  be  displayed,  with  different 
hues/saturation/shading  representing  the  respective  points  that  can  and  have  been  explored,  so 
the  user  can  get  a  sense  of  the  whole  epoch  space  (Goal  #3).  Enabling  dragging  over  several 
epochs  to  select  them  all  would  increase  selection  efficiency  as  well. 

Other  comments 


If  there  are  more  than  two  epoch  variables  to  display,  each  epoch  will  not  necessarily  have  unique 
locations  on  a  scatter  plot  (i.e.,  multiple  different  epochs  will  overlap  and  be  indistinguishable). 
Even  if  more  dimensions  are  encoded  by  size  (area  placing  fourth  on  Cleveland  and  McGill's  list), 
color  (placing  sixth),  and  shape,  as  in  a  bubble  chart,  points  representing  epochs  will  still  be  on 
top  of  each  other  in  x-y  space,  so  users  will  not  have  a  clear  way  of  separating  them  spatially  to 
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find  and  select,  or  to  know  how  deep  the  epoch  space  actually  goes,  impeding  Goal  #1  and  failing 
Goals  #2  and  #3. 


Parallel  Coordinate  Plots 
Description 

Parallel  coordinate  plots,  as  shown  in  Figure  17,  display  high-dimensional  data  by  representing 
each  variable  on  a  vertical  axis  (in  Figure  17,  these  variables  are  "Sepal  Width,"  "Sepal  Length," 
"Petal  Width,"  and  "Petal  Length")  that  are  not  necessarily  scaled  the  same.  An  individual 
("horizontal")  line  spanning  the  axes  represents  the  point  that  takes  the  values  of  each  variable 
it  intersects.  For  example,  following  the  red  line  at  the  top  of  the  "Sepal  Width"  axis,  the 
corresponding  entry  seems  to  have  the  following  approximate  values:  Sepal  Width  -  4.4,  Sepal 
Length  -  5.8,  Petal  Width  -  0.4,  Petal  Length  -  1.5.  Additional  characteristics  can  be  encoded  in 
color,  as  in  Figure  17,  but  are  not  necessary. 


4.5 

4 

3.5 

3 

2.5 

2 

Sepal  Width  Sepal  Length  Petal  Width  Petal  Length 
—  setosa  —  versicolor —  virginica 

Figure  17.  Example  of  Parallel  Coordinate  Plot,  from  1791 
Example  implementation  and  evaluation  of  goals 

Parallel  coordinate  plots  are  quite  effective  at  representing  and  revealing  patterns  in  high¬ 
dimensional  data  when  each  data  point  has  slightly  different  values.  However,  since  epochs  can 
be  generated  as  a  full  factorial  of  epoch  variable  combinations,  many  share  more  than  one 
variable  value,  causing  this  representation  to  suffer  from  the  same  spatial  ambiguity  problem  as 


79  en.wikipedia.org/wiki/Parallel_coordinates 
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greater-than-two-dimensional  scatter  plots.  In  other  words,  the  enumeration  of  epoch  variables 
causes  each  segment  between  adjacent  axes  to  be  shared  among  many  epochs,  again  hiding 
epochs  that  share  the  same  segment,  or  location,  from  the  viewer. 


An  example  sketch  of  this  is  shown  in  Figure  18,  using  the  five  epoch  variables  from  the  Next 
Generation  Combat  Ship  (NGCS)  dataset  developed  by  Schofield.  These  epoch  variables  are 
VUAV,  SmallBoatSize,  EngineEmissions,  Rangelncrease,  and  IceRegionllse  (Schofield  2010; 
Schaffner  2014)80'81.  As  an  example,  two  epochs  that  share  the  same  values  for  VUAV  and 
SmallBoatSize  will  share  their  first  segment,  but  viewers  would  not  be  able  to  tell  that  the 
segment  encoded  more  than  one  entry,  skewing  interpretation  of  the  epochspace.  For  this 
reason,  parallel  coordinate  plots,  as  with  higher-dimension  scatter  plots,  impede  Goal  #1  and  fail 
Goals  #2  and  #3. 

Other  comments 


It  is  worth  noting  that  while  parallel  coordinate  plots  do  not  work  well  as  a  visualization  technique 
for  epoch  sampling,  they  could  be  quite  useful  in  depicting  design  alternatives  for  multi-epoch  or 
-era  analyses  when  the  number  and  variability  of  designs  is  so  high  that  patterns  may  emerge  in 
the  groups  of  design  lines.  Additionally,  modifications  to  the  standard  parallel  coordinate  plots 
have  been  done  in  order  to  overcome  some  of  the  aforementioned  shortcomings.  As  an 
example,  line  segment  width  can  be  used  as  an  indication  of  number  of  overlapping  segments 
(e.g.,  a  thicker  line  segment  indicates  more  overlapping).  Further  modifications,  leveraging  user 
interactivity,  will  be  discussed  in  a  later  section  below. 


80  Schofield,  D.M.  "A  framework  and  methodology  for  enhancing  operational  requirements  development:  Unites 
States  Coast  Guard  cutter  project  case  study."  Massachusetts  Institute  of  Technology,  2010. 

81  Schaffner,  M.A.,  "Designing  Systems  for  Many  Possible  Futures:  The  RSC-based  Method  for  Affordable  Concept 
Selection  (RMACS),  with  Multi-Era  Analysis,"  Master  of  Science  Thesis,  Aeronautics  and  Astronautics,  Massachusetts 
Institute  of  Technology,  June  2014 
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Trees 


Description 

The  next  three  visualization  techniques  require  epoch  variable  data  to  be  stored  in  a  tree 
structure  to  pass  into  built-in  D3.js  layouts.  Trees  make  it  fairly  straightforward  to  visualize  all  of 
the  options  at  each  variable,  making  this  visualization  technique  very  useful  for  hierarchical  data. 
To  find  the  characteristics  of  a  certain  leaf  node  (at  the  bottom  of  a  tree),  one  traverses  up  the 
path  to  the  root  from  the  leaf.  Similarly,  to  find  a  node  with  specified  values  for  each  variable, 
traverse  down  the  corresponding  paths  from  the  root  to  reach  that  node.  An  example  diagram 
of  an  unlabeled  tree  visualization  is  shown  in  Figure  19.  Each  branching  point  represents 
alternative  enumeration  values  for  the  variable  at  a  given  level  of  the  tree. 


Figure  19.  Example  unlabeled  tree  visualization,  from  1821 
Example  implementation  and  evaluation  of  goals 

In  our  implementation,  we  organize  the  data  so  that  each  epoch  variable  is  a  fixed  depth  into  the 
tree,  and  the  epochs  are  the  leaves.  One  such  implementation,  using  the  five  epoch  variables 
from  the  NGCS  database  is  shown  in  Figure  20.  In  this  implementation,  clicking  nodes  toggles 
the  visibility  of  their  children/descendants.  Filled  in  blue  circles  signify  that  the  node  contains 
hidden  children,  and  white  circles  signify  that  the  node  has  been  expanded.  At  each  node,  the 
variable  name  and  value  is  displayed.  To  select  epochs,  the  user  must  click  into  'SELECT  mode/ 
in  which  clicking  nodes  adds  all  descendant  epochs  to  the  list  of  selected  epochs  (e.g.  Clicking 
Epoch  #8's  parent  node  'IceRegionllse:  high'  would  only  select  Epoch  #8,  whereas  clicking  the 
root  node  'All  Epochs'  would  select  all  108  enumerated  epochs).  The  variables'  levels  in  the  tree 


82  https://littleml.files.wordpress.com/2012/01/screen-shot-2012-01-23-at-10-00-17-aml.png 
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can  be  reordered  for  easier  mass  selection  (e.g.,  if  I  only  want  to  select  epochs  with  IceRegionllse: 
high,  I  can  reorder  IceRegionllse  to  be  at  the  top  of  the  tree,  and  only  expand  out  that  node). 


Click  on  the  nodes  to  expand  them. 
Selected  Epochs:  none 
Switch  to  SELECT  mode 


Figure  20.  Example  of  IEEA  Epoch  Selection  on  NGSC  data  implemented  as  a  Tree 

Evaluating  this  interface  with  regard  our  epoch  sampling  goals,  it  does  present  a  clearly  defined 
pathway  to  every  epoch,  so  users  should  easily  be  able  to  tell  how  epochs  are  defined  and  how 
to  find  and  select  epochs  based  on  epoch  variable  levels  (meeting  Goals  #1  and  #2).  If  all  of  the 
nodes  were  expanded,  the  user  would  be  able  to  see  the  size  of  the  full  epoch  space,  and  further 
hue/saturation/shading  could  indicate  the  epochs  the  user  is  able  to  and  already  has  explored 
(meeting  Goal  #3). 


Treemaps 

Description 

Treemaps  are  another  built-in  D3.js  layout  in  which  tree  nodes  are  represented  by  rectangles, 
and  "parent"  rectangles  are  recursively  partitioned  into  smaller  "children"  rectangles.  Again,  this 
is  very  effective  for  hierarchical  data,  as  well  as  representing  all  of  a  tree's  leaf  nodes  compactly 
(Wang  2006)83.  There  is  the  option  of  encoding  additional  values  in  shapes'  color  and  size  to 
reveal  more  attributes  of  leaf  nodes,  but  this  capability  is  not  necessary  for  simply  viewing  the 
whole  epoch  space.  Treemaps  are  generally  good  for  representing  trees  when  the  distribution 
of  children  is  non-uniform,  so  that  they  can  display  the  variety  at  a  glance.  An  example  of  such  a 
treemap  is  shown  in  Figure  21,  displaying  the  populations  of  all  the  countries  in  the  world.  The 
tree  structure  in  this  example  stores  the  six  continents  as  the  children  of  the  root,  and  each 
continent's  children  are  all  the  countries  that  belong  to  that  continent.  The  divisions  between 
continents  in  the  figure  are  denoted  with  bold  black  lines,  whereas  those  between  countries  are 


83  Wang,  Y.,  Teoh,  S.T.,  Ma,  K.  "Evaluating  the  Effectiveness  of  Tree  Visualization  Systems  for  Knowledge  Discovery." 
Eurographics/IEEE-VGTC  Symposium  on  Visualization,  2006. 
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simply  gray.  In  this  example,  the  country's  population  is  encoded  in  area  and  its  Gross  National 
Income  (GNI)  is  encoded  in  color. 


population 


l_  ,  -  [—  ~n=  I,  .  3^ 

0  10000  20000  30000  40000  50000  60000  70000  80000  90000 

GNI 

Figure  21.  Example  treemap  of  country  population  by  continent,  from  1841 
Example  implementation  and  evaluation  of  goals 

In  the  case  of  epoch  sampling,  since  all  sibling  nodes  contain  a  copy  of  the  exact  same 
descendants,  the  treemap  can  be  very  repetitive  and  boring,  as  seen  in  Figure  22,  which  again 
uses  the  aforementioned  NGCS  epochs.  It  is  difficult  to  distinguish  differences  because  of  the 
nodes'  shared  boundaries,  but  the  root  node  (the  outermost  rectangle)  has  first  been  divided 
into  two  parts  (as  the  first  epoch  variable  in  the  tree,  VUAV,  has  two  values),  then  each  of  those 


84  http://www.eecs.tufts.edu/~rveroy/stuff/GNI2010-treemap.png 


59 


has  been  divided  into  two  parts  (the  second  variable  also  has  two  values),  and  so  on,  down  to 
the  leaves  of  the  tree,  the  epochs. 


Figure  22.  Treemap  visualization  of  NGCS  epochs 


In  terms  of  evaluating  this  visualization,  all  of  the  epochs  (the  smallest  division  of  rectangles)  are 
easily  seen  at  a  glance,  but  their  hierarchy,  and  characteristics,  are  difficult  to  distinguish.  Thus, 
while  our  third  goal  can  be  easily  accomplished  by  different  hue/saturation/shading  to  compactly 
display  the  fraction  of  all  possible  epochs  selected,  static  treemaps  do  not  provide  help  to 
accomplish  Goals  #1  and  #2  at  all. 


Circle  Packing 
Description 

Circle  packing  is  yet  another  built-in  D3.js  layout  in  which  tree  nodes  are  represented  by  shapes. 
In  this  visualization,  children  nodes  are  recursively  packed  into  parent  circles  to  fill  the  area  as 
compactly  as  possible,  again  proving  very  effective  for  hierarchical  data.  Again,  there  is  the 
option  of  encoding  additional  values  in  shapes'  color  and  size,  but  this  capability  is  not  necessary 
for  simply  viewing  the  whole  epoch  space.  Figure  23  shows  an  example  of  a  circle  packing  layout 
(the  original  circle  packing  tutorial  on  the  D3  page,  in  fact),  showing  the  Flare85  class  hierarchy. 
The  largest  (outermost)  circle  represents  the  root  node.  Bigger  circles  encompass  all  their 
children,  which  in  turn  encompass  all  their  children  until  the  tree's  leaves  are  reached  (the 
smallest  circles).  In  this  example,  the  leaves  have  been  colored  orange  while  all  intermediate 
nodes  are  shades  of  blue. 


85  Flare  Data  Visualization  library,  http://flare.prefuse.org/ 


60 


Figure  23.  Example  circle  packing  layout,  from  1861 
Example  implementation  and  evaluation  of  goals 

Figure  24  shows  an  example  implementation,  again  using  the  NGCS  dataset.  Given  a  particular 
hierarchical  sequence  of  epoch  variables,  the  circles  represent  that  sequence  in  decreasing  radius 
of  the  packed  circles.  As  shown  in  the  figure,  VUAV  can  take  on  two  values  and  is  represented  as 
two  circles  in  the  left  pane.  The  next  level  down  the  hierarchy  shows  the  three  levels  of 
EngineEmissions,  represented  by  the  three  circles  contained  within  each  VUAV  circle.  The  right 
pane  shows  a  zoomed  in  version  of  one  of  the  VUAV  circles  to  highlight  the  contained  circles. 


86  http://bl.ocks.org/mbostock/raw/4063530/ 
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Figure  24.  Circle  packing  visualization  of  NGCS  epochs  as  seen  at  different  zoom  levels 


As  with  treemaps,  circle  packing  easily  lends  to  the  accomplishment  of  Goal  #3,  allowing  the  user 
to  get  a  sense  of  the  whole  epoch  space  (as  well  as  how  many  variables  go  into  each  epoch  - 
encoded  by  the  number  of  circle  layers),  as  seen  in  the  left  pane  of  the  figure.  Our  particular 
implementation  of  the  circle  packing  visualization  actually  also  allows  users  to  zoom  to  any 
portion  by  clicking  on  corresponding  circles,  as  seen  in  the  right  pane.  Through  this,  a  user  can 
click  through  to  any  particular  epoch  based  on  the  variable  values  from  higher  levels,  helping 
with  Goal  #2.  Finally,  if  for  some  reason  a  user  is  very  zoomed  in  to  a  particular  epoch  and  wants 
to  understand  the  variables  that  went  into  creating  it,  s/he  can  easily  zoom  out  layer  by  layer  to 
discover  them  (meeting  Goal  #1). 

Other  Comments 


While  both  the  treemaps  and  the  circle  packing  visualizations  provide  the  opportunity  to  view 
the  entire  epoch  space  at  a  glance,  it  is  easier  to  recognize  levels  in  circle  packing,  as  the 
boundaries  for  rectangles  overlap,  whereas  the  boundaries  of  circles  do  not.  It  should  be  noted 
that  the  ability  to  zoom  can  also  be  implemented  on  treemaps,  but  for  the  fully  enumerated 
epoch  data,  as  the  rectangles  are  all  still  the  same  size,  their  shared  boundaries  will  not  make  this 
feature  as  useful  as  it  is  for  circle  packing. 


Evaluation  of  Proposed  Implementations 

This  section  summarizes  the  evaluation  of  the  various  considered  visualizations  techniques  for 
use  in  Epoch  Sampling.  To  review,  the  evaluative  criteria  are  as  follows: 
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1)  Is  the  visualization  good  for  two  epoch  variables? 

2)  Is  the  visualization  good  for  more  than  two  epoch  variables? 

3)  Does  the  visualization  help  the  user  understand  specific  epoch  definitions?  (Goal  #1) 

4)  Does  the  visualization  help  the  user  find  and  select  epochs?  (Goal  #2) 

5)  Does  the  visualization  help  the  user  understand  a)  epoch  space  size,  b)  fraction  available 
to  explore,  and  c)  fraction  already  explored?  (Goal  #3) 

The  three  possible  answers  to  these  questions  are: 

•  "Yes"  -  This  visualization  achieves  the  goal. 

•  "Ok"  -  This  visualization  is  mediocre;  does  not  actively  help  nor  hurt  to  achieve  the  goal. 

•  "No"  -  This  visualization  hinders  the  achievement  of,  or  does  not  achieve,  the  goal. 

Criteria  2-5  are  answered  assuming  there  are  greater  than  two  epoch  dimensions.  Table  1 
summarizes  the  relevant  features  of  the  proposed  implementations  from  the  discussion  above 
in  the  context  of  our  IEEA  Epoch  Sampling  goals.  The  best  alternative,  as  reviewed  for  the  Epoch 
Sampling  module,  for  each  row  is  highlighted. 

Table  1:  Summary  of  characteristics  for  each  visualization,  with  best  alternative  for  each  row  indicated. 


Visualization 

Type: 

Scatter 

Plot 

Parallel 

Coordinates 

Tree 

Treemap 

Circle 

Packing 

Good  for  two 
dims 

Yes 

Ok 

Ok 

Ok 

Ok 

Good  for  multi 
dims 

No 

Yes 

Yes 

Yes 

Yes 

Goal  #1 

(understanding) 

Ok 

Ok 

Yes 

No 

Yes 

Goal  #2  (find  & 
select  epochs) 

Ok 

No 

Yes 

No 

Yes 

Goal  #3  (view 
epochspace/fracs) 

Ok 

No 

Yes 

Yes 

Yes 

As  seen,  for  the  case  of  two  epoch  variables,  the  scatter  plot  is  best  available  option  (note  that 
the  scatter  plot  meets  all  goals  in  the  two-dimensional  case).  For  multiple  dimensions,  the  tree 
visualization  meets  all  three  of  our  defined  goals,  making  it  the  single  best  alternative.  However, 
the  most  promising  visualization  technique  for  epoch  sampling  among  these  choices  seems  to  be 
a  coordinated  combination  of  the  circle  packing  with  the  expandable  tree,  as  the  circle  packing 
surpasses  the  tree  in  its  ability  to  facilitate  Goal  #3,  in  order  to  optimize  the  abilities  of  both  of 
these  visualizations  individually. 


Next  Steps 

Immediate  next  steps  will  be  integrating  aspects  of  scatter  plots,  trees,  and  circle  packing  into  a 
demonstration  implementation,  as  well  as  developing  prototypes  for  the  other  parts  of  IEEA  and 
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evaluating  their  functionality  based  on  user  goals  for  these  modules.  Informal  user  interviews 
may  be  conducted  to  ensure  module  goals  stay  accurate.  While  we  have  been  able  to  evaluate 
functionality  of  various  visualizations  as  described,  future  studies  should  test  them  on  users  from 
the  population  of  decision-makers/analysts  who  would  actually  need  to  use  such  a  tool. 


Usability  and  Future  Considerations 

Functionality  is  only  one  attribute  of  a  system.  Analogous  to  Ricci  and  Schaffner's  concepts  of 
"trust"  and  "truthfulness"  in  a  model  (Ricci  2014)87,  flawless  functionality  of  a  system 
("truthfulness")  does  not  guarantee  usability  ("trust").  This  usability  can  only  be  earned  with 
good  interface  design.  Now  that  we  have  presented  and  evaluated  the  functional  visualization 
techniques  based  on  a  particular  module's  goals,  we  next  consider  the  usability  criteria  for 
evaluating  the  interfaces  that  display  these  visualizations  to  users.  These  criteria  include 
learnability,  efficiency,  and  error-tolerance  (based  on  Miller  2011)88. 


Learnability 

When  evaluating  the  learnability  of  visualizations,  some  questions  to  consider  include: 

•  Is  it  easy  to  learn  at  first? 

•  How  helpful  is  the  interface? 

•  Can  tasks  be  completed  and  mastered  without  outside  help? 

•  Does  it  have  built-in  instructions  or  guidance? 

While  many  pieces  of  technology  were  developed  with  the  assumption  that  users  would  read  a 
manual  or  take  a  class  first,  that  is  increasingly  not  the  case.  More  often  than  not,  users  are  goal- 
oriented,  and  will  learn  to  operate  a  system  by  way  of  exploring  how  to  complete  tasks  (learning 
by  doing)  or  by  seeing  others  complete  a  task  (learning  by  watching).  If  users  need  help  from  the 
system  along  the  way  for  whatever  reason,  the  help  must  be  searchable  and  goal-oriented  in 
order  to  be  most  effective.  As  visual  cues  are  much  easier  to  aid  in  user  memory  (“recognition" 
-  knowledge  in  the  world)  than  no  such  help  (“recall"  -  knowledge  in  the  head),  it  is  important 
that  systems  somehow  help  the  user  rather  than  require  the  user  to  remember  everything  about 
its  operation.  As  mentioned  in  the  previous  section  with  functionality,  consistency  is  important 
within  the  interface  as  well  as  externally  (so  perhaps  users  can  transfer  existing  knowledge  from 
other  applications  to  aid  in  using  this  interface).  Quick,  visible  system  responses  are  also  critical 
so  that  the  user  can  get  immediate  feedback  on  whether  or  not  they  have  actually  done 
something.  If  an  interface  has  multiple  states  or  modes,  these  should  also  be  very  apparent  to 
the  user  (and  their  transitions,  if  applicable).  Finally,  the  interface  should  provide  affordances, 
or  the  ability  of  an  object  to  appear  that  it  can  be  used  in  a  certain  way.  For  example,  a  text  box 


87  Ricci,  N.,  Schaffner,  M.A.,  Ross,  A.M.,  Rhodes,  D.H.,  Fitzgerald,  M.E.,  "Exploring  Stakeholder  Value  Models  Via 
Interactive  Visualization,"  12th  Conference  on  Systems  Engineering  Research,  Redondo  Beach,  CA,  March  2014. 

88  Miller,  R.  6.831  User  Interface  Design  and  Implementation,  Spring  2011.  (Massachusetts  Institute  of  Technology: 
MIT  OpenCourseWare) 
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offers  the  affordance  that  a  user  can  click  into  it  and  type.  Ideally,  an  object's  perceived 
properties  to  the  user  should  match  its  actual  properties,  so  the  user  knows  exactly  what  s/he  is 
to  do  with  the  object  (Miller  2011). 


Efficiency 

When  evaluating  the  efficiency  of  visualizations,  some  questions  to  consider  include: 

•  Once  learned,  is  it  fast  to  use? 

•  How  long  does  it  take  to  complete  common  tasks? 

•  Does  the  interface  feel  efficient  to  users? 

•  Are  there  bottlenecks  or  shortcuts? 

Once  a  user  is  familiar  with  a  system,  s/he  tends  to  group  parts  of  it  in  a  unit  of  memory.  This  is 
called  "chunking,"  and  good  interfaces  should  present  information  in  such  chunks  that  are  easily 
recognizable  by  the  user.  The  interface  should  also  be  fast  to  navigate,  in  terms  of  pointing  and 
steering.  Fitts's  Law,  T  =  a  +  b* *log(D/S+l)  =  RT  +  MT,  represents  the  time  T  it  takes  to  move  your 
hand  to  a  target  of  size  S  and  distance  D,  or  the  reaction  time  RT  plus  the  movement  time  MT. 
This  law  for  pointing  has  many  implications  for  interface  design  to  speed  up  pointing  time,  such 
as  the  fact  that  targets  at  the  edge  of  the  screen  are  easy  to  hit,  whereas  unclickable  margins 
require  increased  accuracy.  To  aid  with  pointing  efficiency,  it  is  good  to  make  frequently-used 
targets  bigger  and  put  them  near  each  other.  There  is  a  similar  law  for  steering,  T  =  a  +  b*D/S, 
representing  the  time  T  that  it  takes  to  move  your  hand  through  a  tunnel  of  length  D  and  width 
S.  The  index  of  difficulty,  represented  by  the  constant  b,  is  now  linear  instead  of  logarithmic, 
showing  that  steering  is  much  harder  than  pointing.  Thus  things  like  requiring  the  user  to  steer 
through  narrow  tunnels  on  the  screen  will  severely  damage  efficiency.  Keyboard  shortcuts  or 
anticipating  the  user's  next  movement  (e.g.,  autocomplete)  also  help  users  to  perform  tasks 
faster  (Miller  2011). 


Error-Tolerance 

When  evaluating  the  error-tolerance  of  visualizations,  some  questions  to  consider  include: 


•  Are  errors  few  and  recoverable? 

•  Does  the  design  help  to  prevent  errors? 

•  Does  it  help  when  errors  occur? 

Human  error  is  unavoidable.  Slips  (failure  of  execution)  and  lapses  (failure  of  memory)  are  fairly 
common  simply  due  to  inattention,  but  interfaces  should  take  measures  to  prevent  complete 
mistakes  (using  the  wrong  procedure  for  a  goal).  Some  ways  of  accomplishing  this  are  avoiding 
actions  with  similar  descriptions,  avoiding  habitual  action  sequences  with  identical  prefixes, 
and/or  adding  confirmation  dialogs,  clearly  marked  exits,  manual  overrides,  error  messages  or 
the  ability  to  undo  (Miller  2011). 
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Future  Considerations 


The  marriage  of  functionality  and  usability  does  not  necessarily  cause  user  satisfaction,  just  as 
with  truthfulness  and  trust,  but  ensuring  that  these  questions  are  addressed  by  any  interface  will 
greatly  boost  the  chances.  Immediate  next  steps  in  this  area  will  be  to  evaluate  interfaces 
holistically  on  these  criteria,  conducting  informal  user  studies  if  possible.  As  with  functionality, 
usability  questions  can  be  answered  subjectively  for  any  interface,  but  for  more  thorough  and 
accurate  evaluations,  future  efforts  should  aim  to  conduct  user  studies  drawing  from  the 
population  at  which  this  tool  is  directed. 


Additional  Interactive  EEA  Visualization  Prototypes 

Collaborative  web-based  tools  similar  to  the  IEEA  demonstration  prototypes  described  in  the 
following  sections  are  not  a  new  concept  and  have  previously  been  discussed  in  works  by  Heer89 
and  in  applications  specific  to  engineering  design  by  Liu90.  As  noted  previously,  Spero91 
performed  a  holistic  review  of  81  existing  tradespace  exploration  tools  and  found  wide  variability 
in  the  implementations  and  types  of  functions  performed  by  various  existing  tools.  The 
Framework  for  Assessing  Cost  and  Technology  (FACT),  currently  in  use  by  the  U.S.  Marine  Corps 
Systems  Command  (MCSC),  is  the  only  extensively  developed,  web-based  systems  engineer 
implementation  reviewed  by  Spero  and  has  been  described  in  detail  in  other  literature92'93'94. 
FACT  is  a  sophisticated  tool  for  performing  tradespace  exploration  that  allows  designers  to 
explore  tradeoffs  in  system  attributes  for  user-selected  restrictions  on  design  variables  and 
performance  variable  ranges.  The  prototypes  described  here  provide  a  proof  of  concept  for 
additional  capabilities  for  IEEA  that  specifically  consider  multiple  stakeholder  needs  and  future 
changes  in  context  and/or  mission. 


89  Heer,  J.  and  Agrawala,  M.,  "Design  Considerations  for  Collaborative  Visual  Analytics,"  Information  Visualization, 
7(l):49-62,  2007. 

90  Liu,  Xiaoqing  Frank,  Samir  Raorane,  and  Ming  C.  Leu.  2007.  "A  Web-Based  Intelligent  Collaborative  System  For 
Engineering  Design,"  In  Collaborative  product  design  and  manufacturing  methodologies  and  applications,  pp.  37-58. 
Springer  London. 

91  Spero,  E.,  Bloebaum,  C.,  German,  B.,  Pyster,  A., and  Ross,  A.,  "A  Research  Agenda  for  Tradespace  Exploration  and 
Analysis  of  Engineered  Resilient  Systems,"  13th  Conference  on  Systems  Engineering  Research,  Redondo  Beach,  CA, 
March,  2014. 

92  O'Neal,  M.,  Ender,  T.,  Browne,  D.,  Bollweg,  N.,  Pearl,  C.J.,  and  Brico,  J.,  "Framework  for  Assessing  Cost  and 
Technology:  An  Enterprise  Strategy  for  Modeling  and  Simulation  Based  Analysis."  MODSIM  World  2011  Conference 
and  Expo,  Virginia  Beach,  VA,  October  14. 

93  Ender,  T.,  Browne,  D.,  Yates,  W.,  and  O'Neal,  M.,  "FACT:  An  M&S  Framework  for  Systems  Engineering."  The 
Interservice  /  Industry  Training,  Simulation  &  Education  Conference  (l/ITSEC),  vol.  2012,  no.  1.  National  Training 
Systems  Association. 

94  Browne,  D.,  Kempf,  R.,  Hansen,  A.,  O'Neal,  M.,  and  Yates,  W.,  "Enabling  Systems  Modeling  Language  Authoring  in 
a  Collaborative  Web-based  Decision  Support  Tool."  Procedia  Computer  Science  16:  373-382,  2013. 
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Earth-Imaging  Satellite  Constellation  Case  Study 


Preliminary  work  exploring  techniques  from  visual  and  big  data  analytics  with  applicability  to  EEA 
has  been  investigated  through  application  to  a  previously  developed  case  study.  The  case  study 
implements  parametric  models  of  an  Earth-imaging  satellite  constellation  to  analyze  trades  in 
performance  and  cost.  Design  variables  such  as  number  of  satellites  per  orbital  plane,  number 
of  planes,  optics  size,  and  altitude  are  evaluated  against  measures  of  performance  such  as  optical 
resolution,  target  revisit  time,  percent  global  coverage,  and  lifecycle  cost.  A  summary  of  the 
integrated,  multidisciplinary  system  model  used  for  this  case  study  is  shown  in  Figure  25.  For 
brevity,  details  of  the  case  study  will  not  be  repeated  here,  but  interested  readers  are  referred 
to  the  earlier  paper  for  detailed  descriptions  of  the  case  study  implementation95. 


The  original  case  study,  analyzed  using  traditional  tradespace  exploration  (TSE)  and 
multidisciplinary  optimization  (MDO)  techniques,  was  extended  to  demonstrate  EEA.  To  that 
end,  3  different  system  stakeholders  (military,  commercial  and  earth  science  users),  each  with 
differing  value  functions,  and  2  possible  future  contexts  were  considered.  This  results  in  6  unique 
epochs.  The  lone  context  variable  evaluated  whether  a  disturbance  event  occurs  that  results  in 
the  loss  of  a  percentage  of  the  satellites  within  the  constellation  and  thus  diminished  value 
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Figure  25.  N2  diagram  representing  integrated  multidisciplinary  parametric  models  of  an  Earth  imaging  satellite 

constellation 


Web  BROWSER-BASED  TOOL  IMPLEMENTING  COORDINATED  VISUALIZATIONS 

Implementation  of  an  IEEA  demonstration  needs  to  draw  on  a  combination  of  the  techniques 
described.  This  means  that  IEEA  needs  to  take  into  account  the  practicality  of  representing  large 


95  Curry  M,  La  Tour  P,  Slagowski  S.,  "Multidisciplinary  Design  Optimization  For  A  High-Resolution  Earth-Imaging 
Constellation,"  IEEE  Aerospace  Conf  2015.  Big  Sky,  MT,  March,  2015. 
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amounts  of  data  effectively  given  scarce  communication  resources  (e.g.,  limited  spatial  or 
temporal  resolutions  due  to  hardware  or  software  constraints)96.  Given  the  volume  and 
complexity  of  the  data  that  will  need  to  be  analyzed,  IEEA  methods  and  tools  should  be  capable 
of  providing  data  to  the  decision-maker  in  a  way  that  enhances  cognition.  A  demonstration  of 
the  above  discussed  techniques  for  coordinated  visualization,  OLAP  and  data  reduction  methods 
was  implemented  in  several  prototype  web  browser-based  tools  similar  to  those  described  by 
Sitterle  et  al.97.  To  improve  information  cognition  by  the  user,  guidelines  for  effective 
coordinated  visualization98  and  animated  data  transition99  were  applied. 

Figure  26  below  shows  a  screenshot  of  scatterplot  representations  of  the  utility  (value)  versus 
lifecycle  cost  of  available  design  alternatives.  The  two  scatter  plots  correspond  to  design  values 
evaluated  in  epoch  1,  the  baseline  case,  and  epoch  5  which  represents  a  situation  in  which 
stakeholder  preferences  for  individual  performance  attributes  has  changed.  The  left-hand  plot 
shows  the  utility  versus  cost  of  the  alternatives  evaluated  in  epoch  1.  The  right-hand  plot  shows 
the  same  alternatives  evaluated  in  epoch  5  and  it  is  clear  that  the  resulting  tradespace  has  been 
distorted,  relative  to  epoch  1,  due  to  a  change  in  the  stakeholder  preferences.  To  further  convey 
that  information  to  the  user,  histograms  of  cost  and  utility  are  displayed  with  each  plot. 

If  a  decision-maker  believes  that  it  is  possible  that  the  system  will  experience  both  epoch  1  and 
5  over  the  course  of  its  lifetime,  then  they  might  prefer  a  design  to  be  Pareto  efficient  in  both 
epochs.  However,  since  the  shape  of  the  tradespace  has  changed  between  epochs  1  and  5,  a 
decision-maker  should  not  necessarily  expect  designs  that  were  previously  on  the  Pareto  front 
to  remain  there  in  the  new  epoch.  Applying  the  concepts  of  brushing  and  linking  between 
coordinated  visualizations,  the  user  can  interactively  draw  a  lasso  around  Pareto  efficient  designs 
of  interest  in  epoch  5  and  receive  immediate  feedback  on  where  those  same  designs  appear  in 
epoch  1. 


96  Keim  D.,  "Designing  Pixel-Oriented  Visualization  Techniques:  Theory  And  Applications,"  IEEE  TVCG  (2000);  6(1): 
59-78. 

97  Sitterle  V,  Curry  M,  EnderT,  Freeman  D.,  "Integrated  Toolset  And  Workflow  For  Tradespace  Analytics  In  Systems," 
INCOSE  Int'l  Symp  2014.  Las  Vegas,  NV,  2014. 

98  Scherr  M.,  "Multiple  And  Coordinated  Views  In  Information  Visualization,"  Media  Informatics  Advanced  Seminar 
on  Info  Vis.  2008/2009. 

99  Heer  J,  Robertson  G.,  "Animated  Transitions  In  Statistical  Data  Graphics,"  IEEE  Trans  on  Vis  and  Comp  Graphics 
(2007);  13(6):  1240-1247. 
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Figure  26.  Web-based  tool  showing  coordinated  scatter  plots  and  histograms 


Figure  27.  Brushing  to  filter  designs  across  coordinated  views 

As  shown  in  Figure  27,  all  of  the  coordinated  visualizations  (e.g.,  epoch  1)  are  updated 
simultaneously  to  reflect  only  the  designs  selected  through  brushing  (in  epoch  5).  It  is  clear  from 
the  combined  visualizations  that  while  the  selected  points  are  Pareto  efficient  in  epoch  5,  some 
of  them  deviate  from  the  Pareto  front  in  epoch  1.  User  cognition  of  this  conclusion  is  reinforced 
by  the  utility  histograms  to  the  right  of  each  scatter  plot  that  show  that  the  tight  distribution  of 
utilities  in  epoch  5  are  now  more  spread  out  in  epoch  1.  In  addition  to  brushing,  a  user  can  also 
interactively  filter  data  by  clicking  on  the  histogram  bars  to  effectively  filter  out  all  but  a  selected 
slice  of  the  data.  As  shown  in  Figure  28,  by  clicking  on  the  y-axis  histogram  of  the  right  hand 
figure,  data  not  associated  with  those  bars  is  grayed  out  in  the  coordinated  views. 

Enhanced  understanding  of  the  impacts  of  a  decision  on  multiple  epochs  can  be  very  powerful  as 
demonstrated  in  this  example.  Much  of  that  power  is  driven  by  the  rapid  response  between 
visualizations  provided  to  the  user  interacting  with  them.  As  previously  discussed,  OLAP 
techniques  can  be  applied  to  slice,  dice,  drill  down,  roll  up  or  compute  pivots  of  the  hyper¬ 
dimensional  data  cube  representing  design  alternatives  over  epochs  and  eras.  For  the  example 
presented  here.  Crossfilter,  a  JavaScript  library  which  functions  like  a  client-side  OLAP  server,  has 
been  used  to  allow  rapid  filtering  between  scatterplot  views  and  to  accelerate  grouping  of  the 
thousands  of  data  points  into  the  aggregated  histogram  views.  Latency  between  user 
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interactions  with  any  visualization  and  the  resulting  updates  in  corresponding  visualizations  is  on 
the  order  of  milliseconds.  This  provides  a  seamless  interactive  experience,  which  should  facilitate 
improved  user  cognition  of  the  data  on  which  they  will  base  their  decisions. 


Figure  28.  Coordinated  views  using  histogram  bin  selection  to  slice  data  using  OLAP 


As  the  number  of  design  alternatives  and  epochs  grow,  interactive  coordinated  visualizations  can 
slow,  as  processing  and  rendering  of  the  data  becomes  the  limiting  factor.  Data  reduction 
techniques  can  be  applied  in  these  situations  to  keep  the  large  amounts  of  data  from  becoming 
unduly  burdensome.  Some  past  examples  of  EEA  case  studies  utilizing  filtering  or  sampling 
approaches  include  applications  in  the  transportation100  and  space  domains101.  Since  these 
approaches  have  the  possible  implication  of  concealing  important  information,  we  would  prefer 
to  use  methods  that  allow  us  to  represent  all  of  the  data.  Binned  aggregation  can  allow  us  to 
accomplish  this  aim. 


100  Nickel  J.,  Using  Multi-Attribute  Tradespace  Exploration  For  The  Architecting  And  Design  Of  Transportation 
Systems,  SM  in  Engineering  Systems.  Cambridge,  MA:  MIT,  2010. 

101  Roberts,  C.,  Richards,  M.,  Ross,  A.,  Rhodes  DH,  Hastings  DE.,  "Scenario  Planning  In  Dynamic  Multi-Attribute 
Tradespace  Exploration,"  3rd  Annual  IEEE  Sys  Conf.  Vancouver,  Canada,  March  2009. 
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In  the  prior  examples,  histograms  were  a  type  of  binned  aggregation,  since  they  reduced  the 
larger  set  of  points  to  a  smaller  set  of  rectangular  bars  reflecting  the  amount  of  data  in  each  bin. 
This  required  projection  of  the  2-D  data  into  a  1-D  space.  To  allow  a  decision-maker  to  more  fully 
appreciate  the  underlying  features  of  the  tradespace,  we  would  ideally  like  to  represent  the  2-D 
data  with  fewer  polygons,  while  simultaneously  not  reducing  the  number  of  dimensions.  One 
technique  for  accomplishing  this  is  to  group  data  into  rectangular  bins  and  encode  the  density  of 
points  using  color  hue102'103.  Some  researchers  have  argued  that  hexagonal  bins  can  better 
represent  data  over  rectangular  bins,  to  aid  a  user's  interpretation104.  A  key  rationale  is  the  fact 
that  hexagons  have  more  sides  and  thus  look  more  like  circles,  while  providing  a  regular 
tessellation  of  a  2-D  surface.  Implementing  the  hexagonal  binning  approach  on  the  running 
example  significantly  reduces  the  number  of  polygons  required,  and  thus  speeds  up  interactive 
rendering.  A  screenshot  of  the  example  implemented  with  hexagonal  binning  is  shown  in  Figure 
29. 


Interaction  with  Large  Datasets 

The  previous  interactive  example  provides  a  relatively  simple  demonstration  of  the  usefulness  of 
coordinated  visualizations  and  OLAP  techniques  for  analyzing  EEA  problems.  However,  realistic 
problems,  like  those  currently  posed  by  the  DoD  as  part  of  the  ERS  research  initiative,  will  likely 
require  the  analysis  of  large  numbers  of  designs,  epochs  and  eras  to  provide  insights  to  decision 
makers.  A  logical  next  question  is  therefore,  "Do  these  interactive  coordinated  viewing 
techniques  scale  to  larger  data  sets?".  To  test  scalability,  an  interactive  application  with  10 
coordinated  displays  was  developed.  The  application,  shown  in  Figure  30  and  Figure  31,  was 
tested  using  the  previously  described  satellite  constellation  case  study  for  a  multi-epoch  analysis 
case  using  almost  a  quarter  million  design/epoch  pairs.  This  is  several  orders  of  magnitude  larger 
than  the  previous  example  that  only  considered  on  the  order  of  10,000  design/epoch  pairs  and 
also  significantly  larger  then  EEA  case  studies  described  in  previous  literature. 


102  Liu  Z,  Jiang  B,  Heer  J.,  "ImMens:  Real-Time  Visual  Querying  Of  Big  Data,"  Eurographics  Conference  on  Visulization 
(EuroVis).  2013. 

103  Cleveland  W,  McGill  R.,  "The  Many  Faces  Of  A  Scatterplot,"  Journal  of  the  American  Statistical  Association  (1984); 
79(388):  807-822. 

104  Carr  D,  Littlefield  W,  Littlefield  J.,  "Scatterplot  Matrix  Techniques  For  Large  N,"  Journal  of  the  American  Statistical 
Association  (1987);  82(398):  424-436. 
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Figure  30.  Interactive  Application  with  Multiple  Coordinated  Views 

This  application  displays  a  summary  of  several  key  design  variables  in  histogram  charts  on  the 
top  row.  The  middle  row  contains  charts  summarizing  breakdowns  of  each  system  user  (e.g. 
military,  commercial  and  earth  science),  lifecycle  cost  for  each  design  alternative,  and  the  multi¬ 
attribute  utility  (MAU)  of  each  design/epoch  pair.  The  third  row  provides  a  histogram  summary 
of  the  Fuzzy  Pareto  Number  (FPN)  of  each  design/epoch  pair.  The  FPN  of  a  particular 
design/epoch  pair  can  be  thought  of  as  a  percentage  distance  away  from  the  Pareto  Front105. 
Thus,  as  designers,  we  would  ideally  want  a  design  to  have  an  FPN  of  zero,  indicating  that  the 
design  is  on  the  Pareto  Front,  across  all  epochs. 


105  Smaling,  R.,  "Fuzzy  Pareto  Frontiers  in  Multidisciplinary  System  Architecture  Analysis",  10th  AIAA/ISSMO 
Multidisciplinary  Analysis  and  Optimization  Conference,  Albany,  NY,  September  2004. 
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Much  like  the  previous  example  application  described  above,  this  application  allows  the  designer 
to  apply  filters  to  the  data  set  by  clicking  on  histogram  bars,  dragging  a  filter  window  across  the 
bars  or  clicking  on  a  section  of  the  pie  chart.  A  filter  action  within  any  particular  chart 
automatically  links  to  the  other  charts  and  updates  them  according  to  show  only  the  filtered  in 
data.  The  bottom  row  of  the  application  provides  a  data  table  that  summarizes  the  designs 
remaining  within  the  data  set  after  filters  have  been  applied.  The  example  shown  in  in  the  figure 
shows  filters  applied  to  design  variables  for  protection  (e.g.  shielding  or  maneuvering  propellant), 
epoch  variables  for  user  preference,  and  overall  system  performance  as  quantified  by  lifecycle 
cost,  utility  and  FPN.  The  interpretation  of  this  example  is  that  the  designer  is  leaving  filtered  in 
only  designs  that  have  no  shielding,  corresponding  to  only  the  military  system  user,  system 
lifecycle  costs  below  $1  billion,  utility  scores  above  0.5  and  FPN  below  10%.  This  filters  the 
number  of  designs  under  consideration  quickly  down  from  242,569  to  only  727  alternatives  as 
shown  in  the  summary  table  at  the  bottom.  After  application  of  the  filters,  updates  of  all  charts 
and  table  elements  take  place  in  ~100  milliseconds  which  is  generally  regarded  as  sufficiently  fast 
enough  to  maintain  user  engagement  while  they  interact  with  the  large  data  set. 
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Figure  31.  Interactive  Application  with  Coordinated  Views  showing  use  of  data  filters 


Advanced  Visualizations  for  Single  and  Multi-Epoch  Analysis 

The  previously  described  interactive  prototypes  provide  good  demonstrations  of  some  of  the 
capabilities  needed  for  IEEA  such  as  multiple  coordinated  views,  OLAP  and  data  reduction 
techniques  such  as  filtering  and  binned  aggregation.  Some  exploratory  work  has  also  been 
performed  on  more  compact  coordinated  visualizations  that  may  improve  the  speed  with  which 
a  designer  can  extract  insights  from  data.  As  distances  between  user  interface  (Ul)  items  on  the 
screen  get  further  apart  the  time  it  takes  to  interact  with  them  should  go  up,  thus  slowing 
discovery  and  decision-making.  Fitt's  Law,  a  descriptive  model  of  human  movement  primarily 
used  in  human-computer  interaction  (HCI)  research,  tells  us  that  interaction  time  will  go  up 
logarithmically  with  increasing  distance  of  required  movement  across  the  Ul  on  the  screen  (Fitts 
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1954)106.  The  interaction  time  is  also  a  function  of  the  size  of  the  Ul  item,  such  as  a  button,  slider 
or  data  point  on  a  graph.  This  is  true  for  increasing  distance  between  mouse  clicks  on  controls 
and  for  eye  movement  between  complementary  Ul  charts  or  visualizations  that  a  designer  must 
use  concurrently  to  reach  a  conclusion.  Another  important  Ul  design  concept  often  used  in  HCI 
work  is  the  steering  law.  The  steering  law  can  be  derived  directly  from  Fitt's  law  and  describes 
the  amount  of  time  it  takes  a  user  to  move  along  a  defined  path  using  a  pointing  device,  like  their 
eyes,  finger  or  mouse  cursor  (Rashevsky  1959)107,  (Drury  1971)108.  This  is  an  important  concept 
to  consider  when  a  user  is  expected  to  interact,  for  instance,  with  nested  menus,  scroll  bars  or 
pop-up  context  menus. 

As  research  on  IEEA  progresses,  these  and  other  Ul  design  concepts  will  be  considered  when 
developing  future  demonstrations.  A  set  of  metrics  that  evaluate  the  usability  and  effectiveness 
of  a  particular  prototype  application  for  decision-making  may  also  be  developed  using  Ul  design 
concepts  such  as  affordance,  efficiency,  safety  and  learnability.  That  a  user  interface  should  be 
safe  and  easy  to  learn  may  seem  obvious,  but  they  can  be  difficult  concepts  to  measure.  The 
affordances  of  a  Ul,  which  refers  to  an  attribute  of  an  object  that  allows  people  to  know  what  do 
with  it,  may  similarly  be  difficult  to  evaluate.  An  example  of  the  concept  of  affordance  would  be 
the  handles  on  a  door  that  tell  a  person  at  a  quick  glance  whether  they  should  push  or  pull  to 
open  it.  Efficiency  may  be  the  most  straight-forward  of  these  concepts  and  refers  to  how  quickly 
a  user  can  interact  with  a  Ul  to  accomplish  a  task.  Each  of  these  concepts  provides  important 
usability  considerations  that  facilitate  improved  collaboration  between  a  human  decision-maker 
and  an  interactive  computer  application  designed  to  aid  them.  A  review  of  recent  literature  in 
the  are  of  HCI  is  necessary  to  determine  the  most  effective  way  of  evaluating  metrics  for  these 
concepts.  Note  that  usability  is  a  concept  distinct  from  the  effectiveness  or  usefulness  of  an 
interface.  An  application  can  be  very  usable,  but  still  fail  to  be  useful  for  decision-making.  Both 
the  usability  and  usefulness  of  interface  must  be  considered  in  future  IEEA  research. 

When  visualizing  large  sets  of  multi-dimensional  data  an  often  applied  approach  is  to  use  scatter 
plot  matrices  that  plot  each  variable  against  each  other  variable  in  a  grid  of  scatter  plots109'110'111. 
This  allows  a  designer  to  detect  patterns  in  the  data  that  may  be  difficult  to  train  an  algorithm  to 
detect  and  is  common  in  exploratory  data  analysis.  For  data  of  high-dimensionality,  like  the  data 
sets  that  may  be  required  for  IEEA,  this  may  require  increasingly  larger  amounts  of  screen  "real- 
estate"  that  may  slow  interaction.  Alternative  visualizations,  such  as  the  one  shown  in  Figure  32, 
may  be  able  to  accomplish  similar  functionality  to  scatter  plot  matrices  with  reduced  interaction 


106  Fitts,  P.,  "The  information  capacity  of  the  human  motor  system  in  controlling  the  amplitude  of  movement". 
Journal  of  Experimental  Psychology  47  (6):  381-391,  June  1954. 

107  Rashevsky,  N.,  "Mathematical  biophysics  of  automobile  driving".  Bulletin  of  Mathematical  Biophysics,  21,  375- 
385,  1959. 

108  Drury,  C.,  "Movements  with  lateral  constraint.  Ergonomics",  14,  293-305,  1971. 

109  Liu  Z,  Jiang  B,  Fleer  J.  Immens:  real-time  visual  querying  of  big  data.  Eurographics  Conference  on  Visulization 
(EuroVis).  2013 

110  Sitterle  V,  Curry  M,  Ender  T,  Freeman  D.  Integrated  toolset  and  workflow  for  tradespace  analytics  in  systems. 
INCOSE  Int'l  Symp  2014.  Las  Vegas,  NV,  2014 

111  Scherr  M.  Multiple  and  coordinated  views  in  information  visualization.  Media  Informatics  Advanced  Seminar  on 
Info  Vis.  2008/2009 
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time.  This  particular  example  uses  two  coordinated  views,  a  scatter  plot  and  a  parallel  coordinate 
plot  that  maps  design  variable,  performance  attributes,  cost,  utility  and  FPN  for  each 
design/epoch  pair.  The  interface  also  includes  a  control  panel  below  the  parallel  coordinates  that 
allows  each  of  the  variables  to  be  assigned  to  the  various  data  dimensions  within  the  scatter  plot. 
The  controls  allow  variables  to  be  assigned  to  the  X-Y  position  of  points  on  the  scatter  plot  as 
well  as  the  radius  of  the  dot  and  its  color.  The  interface  presents  the  same  information  as  the 
traditional  scatter  plot  matrix  in  a  much  more  compact  form  that  facilitates  user  interaction,  data 
filtering  and  visualization. 
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Figure  32.  Coordinated  multiple  views  of  single  epoch  results  using  X-Y  scatterplot  and  parallel  coordinates 


As  mentioned,  in  addition  to  the  position  (e.g.  x,y  coordinates)  of  elements  within  the  scatter 
plot,  other  attributes  of  a  visualization  can  also  be  used  to  display  other  data  dimensions.  In 
Figure  33,  additional  data  dimensions  are  encoded  using  color  and  circle  radius.  In  this  example 
x,  y,  radius  and  color  correspond  to  cost,  utility,  aperture  size  and  gap  time  respectively.  The  user 
can  control  which  variables  are  assigned  to  which  visualization  elements  using  the  control  panel 
at  the  bottom.  This  provides  the  user  a  convenient  way  to  visualize  the  highly  dimensional  data 
much  more  quickly  to  gain  insights  and  make  decisions. 
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Figure  33.  Visualization  with  additional  data  dimensions  encoded  using  color  and  circle  radius 


To  gain  a  deeper  understanding  of  the  data,  however,  a  user  needs  to  be  able  to  interactively 
explore  and  filter  the  data.  As  shown  in  Figure  34,  this  prototype  interface  allows  a  designer  to 
interactively  set  filters  by  dragging  a  box  around  values  on  the  axes  of  the  parallel  coordinate 
plot.  In  this  example  filters  are  set  to  keep  FPN  less  than  10%,  cost  less  than  $1B  and  utility 
greater  than  0.5.  The  designs  remaining  are  biased  towards  near  polar  inclination  constellations 
at  high  altitudes,  near  100%  global  coverage  and  low  gap  times.  This  is  a  relatively 
straightforward  example,  but  it  demonstrates  how  a  designer  can  use  more  advanced 
coordinated  visualizations  to  understand  high-dimensional  data.  This  is  an  important  proof  of 
concept  for  capabilities  that  can  be  applied  to  future  IEEA  demonstrations  that  will  require 
analysis  of  high-dimensional  data. 
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Figure  34.  Visualization  showing  applied  filters 


Next  Steps 

Recent  advances  as  well  as  ongoing  efforts  by  various  researchers  and  industry  practitioners 
create  new  opportunities  for  the  development  of  advanced  systems  engineering  methods, 
processes  and  tools.  Developments  in  computing,  visual  analytics  and  statistical  algorithms  all 
provide  techniques  that  could  potentially  be  applied  to  the  field  of  systems  engineering.  This 
could  result  in  better  ways  to  design  complex  systems  to  handle  a  wide  range  of  operational 
contexts  and  future  stakeholder  needs  while  effectively  controlling  the  escalating  costs 
associated  with  acquiring,  operating  and  maintaining  such  systems. 

An  ultimate  goal  for  this  research  effort  is  to  demonstrate  end-to-end  applications  of  IEEA  on 
several  case  studies.  These  case  studies  could  include  both  multi-epoch  and  multi-era  analysis 
and  serve  as  demonstrations  of  how  IEEA  enables  a  decision-maker  to  design  systems  with  the 
ability  to  respond  to  new  or  changing  conditions  through  modified  tactics,  reconfiguration,  or 
replacement.  Leveraging  recent  work  in  visual  analytics  and  coupling  it  with  new  applications  of 
EEA  constructs  using  improved  processes  are  seen  as  enablers  of  this  goal.  IEEA  will  allow 
designers  to  better  inform  the  identification  of  relevant  questions,  uncover  patters,  discover 
regions  of  interest  within  the  tradespace  and  potential  model  errors,  and  ultimately  allow  the 
decision-maker  to  make  better  informed  decisions  with  higher  levels  of  confidence  and  trust. 
Several  areas  of  ongoing  investigation  may  contribute  to  this  end  goal. 
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First,  structured  approaches  for  querying,  manipulating  and  visualizing  the  types  of  large  data 
sets  generated  through  EEA  must  be  matured  further.  As  part  of  this  effort  so  far,  several 
prototypes  have  demonstrated  capabilities  necessary  for  IEEA  such  as  massive  parallelization, 
multiple  coordinated  interactive  visualization  and  OLAP  for  faster  data  handling.  Additional 
research  may  further  extend  our  ability  to  use  EEA  constructs  and  higher  interactive  rates. 
Examples  include  demonstrating  methods  to  allow  a  decision-maker  to  interact  with  their  data 
at  multiple  resolutions  or  levels  of  abstraction,  real-time  surrogate  modeling  and  predictive 
prefetching  of  data  possibly  based  on  user  modeling.  All  of  these  methods  have  the  potential  of 
relaxing  the  requirement  that  an  interactive  system  must  have  all  data  in  memory  at  once.  On 
larger  problems,  like  those  posed  by  the  DoD  ERS  initiative,  it  may  not  be  possible  to  hold  all  data 
associated  with  a  design  problem  in  the  memory  of  a  personal  computer  or  workstation  at  the 
same  time. 

Second,  additional  research  is  necessary  to  refine  the  process  defining  the  IEEA  framework.  One 
potential  key  to  improving  interactive  collaboration  between  a  human  analyst  and  an  interactive 
application  (or  ensemble  of  applications)  is  recognition  of  the  importance  of  the  process  the 
human  follows.  The  interactive  process  may  in  fact  be  more  important  than  the  sophistication 
of  the  application  or  the  experience  of  the  analyst.  Thompson  (2010)112  provides  an  example  of 
human  chess  players  using  simple  computer  tools  to  play  against  a  sophisticated  super  computer 
built  by  IBM.  The  case  study  showed  that  the  super  computer  could  easily  outmatch  a  human 
grandmaster,  but  if  the  human  was  augmented  with  a  basic  software  application  running  on  a 
simple  laptop  they  could  consistently  beat  the  more  powerful  super  computer.  More 
surprisingly,  the  case  study  revealed  that  amateur  players  that  used  a  superior  process  for 
interacting  with  simple  computer  tools  could  beat  both  the  super  computer  and  human 
grandmasters  that  was  also  augmented  with  computer  tools  but  following  an  inferior  interaction 
process.  The  key  take-away  is  that  the  process  through  which  the  human  and  computer  interact 
strongly  influences  the  effectiveness  of  their  collaboration. 

In  addition  to  the  interactive  process,  research  on  improvements  to  the  IEEA  framework  should 
also  investigate  how  to  effectively  apply  iteration  within  the  revised  process.  Noted 
mathematician  John  Tukey  described  exploratory  data  analysis  as  "an  open-ended,  highly 
interactive,  iterative  process"  (Jones,1987)113.  IEEA  is  seen  as  an  extension  of  previous  methods 
like  RSC  (Ross  2009)114  but  with  specific  emphasis  placed  on  a  more  iterative,  interactive  process. 

A  third  and  final  candidate  area  of  research  is  the  development  of  user  models  to  improve 
human-computer  collaboration.  As  also  noted  by  Tukey,  "nothing  can  substitute  here  for  the 
flexibility  of  the  informed  human  mind",  but  user  models  may  be  useful  to  facilitating  a  more 
effective  collaboration  during  design/epoch-space  exploration.  User  models  may  be  useful  when 


112  Thompson,  C.  (2010,  February  1).  Garry  Kasparov,  cyborg.  Retrieved  February  27,  2015,  from 
http://www.collisiondetection.net/mt/archives/2010/02/why_cyborgs_are.php 

113  Jones,  L.,  "The  Collected  Works  of  John  W.  Tukey:  Philosophy  and  Principles  of  Data  Analysis",  1965-1986,  Volume 
4,  CRC  Press,  May  15,  1987. 

114  Ross  A.,  McManus,  H.,  Rhodes  D.,  Hasting,  D.,  and  Long,  A.,  "Responsive  Systems  Comparison  Method:  Dynamic 
Insights  into  Designing  a  Satellite  Radar  System",  AIAA  Space  2009,  Pasadena,  CA,  September,  2009. 
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coupled  with  machine-learningtechniquesto  offer  the  human  analyst  recommendations  on  what 
they  may  be  looking  for  based  on  past  data.  This  can  be  thought  of  in  the  same  way  search 
engines  algorithms  like  those  used  by  Google  and  Amazon  offer  suggestions  based  on  past  data 
and  initial  queries.  User  models  may  also  be  helpful  for  predicting  data  that  must  be  pre¬ 
computed  or  pre-fetched  from  a  remote  database  by  predicting  what  the  user  will  want  to  look 
at  next.  As  previously  discussed,  predictive  approaches  like  these  could  be  particularly  important 
when  working  on  problems  where  it  is  not  possible  to  store  all  design  data  locally  at  the  same 
time. 

Investigations  into  each  of  these  research  areas  should  culminate  in  an  eventual  end-to-end 
demonstration  of  IEEA  on  several  relevant  case  studies.  These  cases  should  clearly  demonstrate 
how  IEEA  empowers  decision-makers  to  make  better  decisions  that  are  either  faster,  more 
trustworthy  and/or  of  a  higher  quality.  They  should  also  demonstrate  how  IEEA  enhances  the 
ability  to  understand  and  communicate  data  through  the  design  of  new  interactive  systems  for 
data  visualization  and  analysis. 

The  research  team  will  mature  the  approach  for  evaluating  systems  under  dynamic  uncertainty, 
with  further  development  of  the  extended  framework  to  for  interactive  capability  and  scaling  to 
big  data.  This  work  extends  the  Phase  2  effort  on  a  demonstration  prototype,  using  the  MIT 
IVTea  Suite,  applying  IMCSE  principles  to  enhance  the  user  interface,  data  handling  and  analysis 
widgets.  In  Phase  3  the  research  team  will  enhance  the  method  and  degree  of  interactive 
capability,  focusing  specifically  on  the  Epoch-Era  Analysis  method,  a  novel  method  for  value- 
driven  tradespace  exploration  and  analysis.  The  maturing  prototype  framework  with  associated 
supporting  tools  will  be  applied  to  a  case  analysis  including  various  types  of  uncertainties.  This 
case  application  will  be  used  to  elicit  feedback  on  relevance,  ease  of  use,  feasibility  and 
tractability  of  data  scaling  and  visualization  techniques.  The  research  team  will  extend  the  Phase 
2  prototype  efforts  for  Interactive  Epoch-Era  Analysis  (IEEA)  and  test  using  case  applications, 
along  with  preliminary  supporting  infrastructure.  This  will  inform  the  transition  strategies, 
additional  case  application  and  prototype  user  testing. 
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Value-Model  Choice  and  Tradeoff 


One  of  the  key  challenges  identified  in  preliminary  research  for  IMCSE  involves  understanding 
the  role  that  model  choice  plays  in  the  generation  and  analysis  of  data  for  decision  making. 
IMCSE  anticipates  making  a  key  contribution  in  terms  of  framing  this  challenge  and  insights 
gained  when  actively  trading  models  as  a  part  of  a  study. 


Introduction 

As  evidenced  by  the  recent  rise  of  model-influenced  systems  engineering  efforts,  including 
Model-Based  Systems  Engineering,  Model-Based  Engineering,  and  Interactive  Model-Centric 
Systems  Engineering,  the  role  of  models  in  engineering  activities  has  been  increasing  in  scope 
(Rhodes  and  Ross  2014)115.  Models  have  always  been  used  as  tools  to  augment  human  ability  to 
make  predictions  or  sense  of  information,  encapsulating  existing  knowledge,  as  well  as 
automating  its  application.  The  rapid  rise  of  low  expense  computational  ability  has  increased  the 
accessibility  of  numerical  models  and  the  roles  they  can  play  in  engineering,  including  both 
analysis  and  synthesis.  Leveraging  models  in  an  effective  way  for  engineering  decision  support 
necessitates  understanding  the  role  that  model  choice  plays  in  the  generation  and  analysis  of 
data  for  decision  making.  This  is  especially  true  when  seeking  to  identify  system  solutions  in  early 
design  that  are  robust  across  uncertainties  (Spero  et  al.  2014)116.  This  report  section  describes 
preliminary  research  in  helping  to  frame  this  challenge  and  potential  insights  that  might  be 
gained  when  actively  trading  models  as  a  part  of  a  study. 


Motivation/Background 

There  are  several  key  concepts  involved  during  design  decision  making  in  early  phase  design. 
Figure  35  depicts  the  general  relationship  between  decision  problems  and  decision  solutions  as 
they  relate  to  data  and  models  in  early  phase  engineering  analysis.  In  this  figure,  decision 
problems  suggest  a  space  of  potential  solutions,  which  span  a  design  space.  The  design  space  is 
then  sampled  and  evaluated  through  two  types  of  models:  cost  models  and  performance  models. 
Cost  models  seek  to  predict  the  resources  needed  to  develop  and  operate  each  of  the  evaluated 
potential  systems.  Typically  these  estimates  are  in  terms  of  dollars,  and  potentially  time  (i.e. 
schedule).  Performance  models  seek  to  predict  the  operational  behavior  in  context  of  the 
evaluated  potential  systems.  Value  models  seek  to  map  the  resulting  resource  and  performance 
predictions  into  decision-friendly  perceived  benefit  and  cost  metrics.  Value  models  can  be  simple 
(e.g.,  just  the  cost  and  performance  measures),  or  complex  (e.g.  aggregate  perceived  benefit  and 
cost  under  uncertainty  of  a  large  number  of  measures),  with  many  possible  implementations  (Lee 
et  al.  2014)117.  Each  of  these  models,  and  the  artificial  data  generated  by  them,  can  be  potentially 


115  Rhodes  DH,  Ross  AM.  Interactive  Model-Centric  Systems  Engineering  (IMCSE)  Phase  1  Technical  Report.  SERC- 
2014-TR-048-1;  September  2014. 

116  Spero  E,  Avera  MP,  Valdez  PE,  Goerger  SR.  Tradespace  exploration  for  the  engineering  of  resilient  systems.  12th 
Confon  Sys  Eng  Research.  (CSER14).  Redondo  Beach,  CA,  March  2014. 

117  Lee  BD,  Binder  WR,  Paredis  CJJ.,  "A  Systematic  Method  For  Specifying  Effective  Value  Models,"  CSER14.  March 
2014. 
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altered  by  changes  in  the  epoch  space  (i.e.,  exogenous  context  and  needs  changes).  Updating 
occurs  when  users  seekto  modify  the  space  definitions,  orthe  models,  in  orderforthem  to  better 
address  the  problem  under  consideration  (or  to  improve  the  trust  or  truthfulness  (i.e.,  validity) 
of  the  models  and  data). 


Decision 
( Problem ) 


Decision 
( Solution ) 


Figure  35.  Role  of  key  models  for  supporting  system  decision  making,  with  alternative  value  models  use  in 

demonstration  case 


Since  the  role  of  models  is  central  in  the  depicted  decision  framework,  it  is  essential  that 
engineers  and  analysts  understand  not  only  the  sensitivities  of  their  proposed  solutions,  but  also 
of  the  models  from  which  the  data  for  decisions  are  generated.  This  includes  understanding  the 
impacts  of  key  assumptions  and  model  formulations  on  the  data.  One  means  for  conducting  this 
investigation  is  through  "model  trading"  (i.e.,  model  selection)  where  data  is  generated  using 
alternative  models  with  the  resulting  data  compared. 


In  this  research,  the  team  has  begun  exploratory  work  defining  model  types  and  formulation  of 
how  model  trading  might  be  implemented.  Leveraging  insights  from  earlier  work  (Ricci  at  al. 
2014)118,  which  described  the  role  of  interactivity  in  refining  a  user's  captured  value  model,  we 
generalize  the  concept  as  "value  model  trading."  This  ranges  from  tuning  parameters  within  a 
particular  value  model  (e.g.,  utility  function  shapes  and  weights  for  a  Multi-Attribute  Utility  value 
model)  to  also  include  trading  of  value  model  formulations  themselves.  There  are  many  possible 


118  Ricci  N,  Schaffner  MA,  Ross  AM,  Rhodes  DH,  Fitzgerald  ME.  Exploring  stakeholder  value  models  via  interactive 
visualization.  CSER14.  March  2014. 
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value  models  (Ross  et  al.  2010)119.  For  this  demonstration,  four  alternative  value  model 
formulations  were  used:  Multi-Attribute  Utility  (MAU),  Analytic  Hierarchy  Process  (AHP),  Cost- 
Benefit  Analysis  (CBA),  and  Measure  of  Effectiveness  (MOE).  Recall,  a  value  model  attempts  to 
predict  how  a  particular  decision  maker  might  perceive  net  benefits  and  costs  for  alternatives 
under  consideration.  Different  value  models  treat  the  mapping  of  raw  data  to  perceived  benefits 
and  costs  differently.  For  illustration  purposes,  we  treated  perceived  costs  as  just  lifecycle  cost 
(essentially  as  a  single  dimensional  metric  of  perceived  cost),  while  we  varied  the  perceived 
benefit  model  across  MAU,  AHP,  CBA,  and  MOE.  The  results  of  this  variation  were  analyzed  in 
terms  of  how  the  set  of  perceived  benefit  versus  cost  efficiency  changed.  This  was  calculated  as 
the  Pareto  efficient  set  (i.e.,  non-dominated  solutions  across  the  two  high  level  objectives)  for 
the  given  value  models.  The  sets  were  then  compared  to  see  the  impact  of  value  model  choice 
on  proposed  "best"  alternative  solutions.  This  demonstration  case  utilized  the  IVTea  Suite 
software  being  developed  internally  at  MIT  to  support  value-driven  tradespace  exploration  and 
analysis. 


Demonstration  of  Value  Model  Trading:  Space  Tug 

For  this  exploratory  case,  the  problem  is  framed  as  the  following: 

A  decision  maker  has  a  budget  for  an  orbital  transfer  vehicle  (a.k.a.  "Space  Tug")  and 
thinks  he  knows  what  he  wants  (in  terms  of  attributes  of  goodness  of  a  system).  But  he 
is  aware  that  he  may  not  have  formulated  his  value  model  correctly.  He  wants  to  explore 
three  types  of  uncertainties  in  his  value  model: 

1.  What  value  model  best  represents  his  preferences? 

2.  What  parameters  for  a  given  value  model  best  represent  his  preferences? 

3.  What  if  he  really  doesn't  know  what  his  true  preferences  are  and  wants  instead  a 
robust  solution? 

The  second  question  was  previously  addressed  (Ricci  et  al.  2014)120,  while  the  first  and  third 
questions  are  investigated  in  this  study.  The  approach  in  this  study  is  to  use  four  different  value 
models  to  evaluate  and  represent  benefit  vs.  cost  tradeoffs;  identify  the  most  value  efficient 
alternatives  under  different  value  models;  compare  preferred  alternatives  across  value  models; 
and  find  solutions  that  perform  well  across  the  alternative  value  models. 


119  Ross  AM,  O'Neill  MG,  Hastings  DE,  Rhodes  DH.  Aligning  perspectives  and  methods  for  value-driven  design.  AIAA 
Space  2010.  Anaheim,  CA,  September  2010. 

120  Ricci  N,  Schaffner  MA,  Ross  AM,  Rhodes  DH,  Fitzgerald  ME.  Exploring  stakeholder  value  models  via  interactive 
visualization.  CSER14.  March  2014. 
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Models  Used  In  The  Case 


The  design  alternatives  and  performance  and  cost  models  for  Space  Tug  are  relatively 
straightforward,  consisting  of  the  rocket  equation  and  some  linear  relationships  (McManus  and 
Schuman  2003)121.  The  value  models  used  in  this  study  are  now  described: 

Multi-Attribute  Utility  (MAU) 

Multi-Attribute  Utility  value  model  generates  an  aggregate  measure  across  multiple  criteria 
(called  attributes)  (Keeney  and  Raiffa  1993)122.  Each  of  the  attributes  have  single  attribute  utility 
functions  that  map  attribute  level  to  perceived  benefit  under  uncertainty  of  that  attribute 
(typically  quantified  on  a  zero  to  one  scale).  The  set  of  single  attribute  utility  functions  is  then 
aggregated  via  a  multi-linear  function  into  a  multi-attribute  utility  score.  The  equation  for  MAU 
is: 

U(X)  =  [nr=i(^r<A(*i)+i)]-^  where  K  =  +  Y\f=1(K  -kt  +  1) 

Here  K  is  the  normalization  constant,  t/(x)  is  the  aggregate  MAU  value  across  the  multiple  single 
attributes  X,  and  their  respective  single  attribute  utilities  t/j(Xi);  k,  is  the  elicited  swing  weighting 
factor  for  attribute  X,;  n  is  the  number  of  attributes.  Figure  36  illustrates  the  three  single  attribute 
utility  functions  (i.e.,  capability,  delta  V,  response  time),  along  with  their  kj  weights  for  the  MAU 
function.  In  the  special  case  where  the  weights  add  to  1,  the  function  becomes  a  linear  weighted 
sum,  and  therefore  each  attribute  contributes  independently  to  the  aggregate  value. 


121  McManus  H.  Schuman  T.  Understanding  the  orbital  transfer  vehicle  trade  space.  AIAA  Space  2003.  Long  Beach, 
CA,  September  2003. 

122  Keeney  RL,  Raiffa  H.  Decisions  with  Multiple  Objectives:  Preferences  and  Value  Tradeoffs.  New  York:  Cambridge 
University  Press;  1993. 
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Figure  36.  Single  attribute  utility  functions  for  the  MAU  value  model. 


Figure  37.  MAU  benefit  vs.  cost  tradespace  (with  Pareto  efficient  set  indicated) 

Each  of  the  Space  Tug  design  alternatives  were  then  evaluated  in  terms  of  the  MAU  benefit  and 
cost  and  are  plotted  in  Figure  37.  Additionally,  the  Pareto  efficient  set  of  designs,  which  are  the 
most  benefit-cost  efficient  solutions,  non-dominated  in  this  two  objective  space,  are  indicated 
with  blue  triangles  (flat  side  on  bottom).  Due  to  the  nature  of  MAU,  design  alternatives  that  do 
not  meet  minimum  acceptable  levels  in  any  particular  attribute  are  deemed  unacceptable  and 
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are  treated  as  infeasible.  This  results  in  a  smaller  set  of  designs  to  consider  (here  as  N=83,  out  of 
the  total  possible  of  384).  The  designs  in  the  Pareto  set  did  not  share  many  common  features, 
but  all  had  propulsion  systems  that  were  electric  (type  3)  or  nuclear  (type  4). 

Analytic  Hierarchy  Process  (AHP) 

Analytic  Hierarchy  Process  value  model  generates  an  aggregate  measure  across  multiple  criteria 
(Saaty  2004)123.  Each  of  the  criteria  are  evaluated  pair-wise  to  determine  relative  value 
contribution.  The  aggregate  AHP  score  is  determined  using  a  linear-weighted  sum,  with  the 
weights  derived  from  the  pairwise  comparisons.  The  AHP  value  equation  is: 

AHP{X)  =  Z?=1  kt  ■  AHPiiXO,  where 

AHPl(Xl)  —  ^.mm)  ^  ij:  jjjgger  /s  better  for  X,;  or  AHPl(Xl)  —  ^X,jnax  XlJ .  jf  smaller  is 

^i,max~^i,min  ^  i.max  ~  ^  i,min 

better  for  Xi, 


*--q=±-£n  a 

kt  — - P=1  P'q,  where  apq  is  the  element  in  row  p,  column  q  in  the  AHP  matrix,  n  is  the  number 

of  criteria. 
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Figure  38.  Matrix  of  comparisons  for  the  AHP  value  model. 


Higher  is  Betters 


Lower  is  Better  ■=> 


Figure  38  illustrates  the  pair-wise  comparison  matrix  for  the  three  criteria  (capability,  delta  V, 
and  response  time),  which  resulted  in  calculated  k,  weights  of  0.4,  0.4,  and  0.2  respectively. 


123  Saaty  TL.  Decision  Making  —  The  Analytic  Hierarchy  and  Network  Processes  (AHP/ANP).  Journal  of  Systems 
Science  and  Systems  Engineering  2004;  13(1):  1-35. 
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Figure  39.  AHP  benefit  vs.  cost  tradespace  (with  Pareto  efficient  set  indicated). 

Each  of  the  Space  Tug  design  alternatives  were  then  evaluated  in  terms  of  the  AHP  benefit  and 
cost  and  are  plotted  in  Figure  39.  Additionally,  the  Pareto  efficient  set  of  designs  are  indicated 
with  green  triangles  (flat  side  on  right).  Due  to  the  nature  of  AHP  value,  no  design  alternatives 
are  rejected,  so  the  full  tradespace  appears  feasible  (N=384).  The  designs  in  the  Pareto  set  have 
no  obvious  pattern  except  they  never  have  electric  propulsion  (type  3). 

Cost-Benefit  Analysis  (CBA) 

Cost-Benefit  Analysis  value  model  converts  multiple  criteria  into  a  common  currency  (typically 
dollars)  in  order  to  simplify  comparisons  (Mishan  2007)124.  In  order  to  construct  this  model,  one 
must  create  monetization  (conversion)  functions  for  each  of  the  criteria.  For  this  case 
demonstration,  each  conversion  function  has  three  parameters,  which  assumes  a  minimum 
acceptable  level  (zero),  a  marginal  dollar  per  unit  of  the  attribute  (the  conversion  rate),  and 
(optionally)  a  diminishing  returns  rate  (if  the  marginal  rate  decreases  with  an  increase  in  attribute 
level).  After  calculating  each  individual  criterion  as  a  dollar  figure,  the  aggregate  is  a  simple  sum 
of  the  three.  The  equation  for  CBA  value  is: 

CBA(X)  =  JZ=1CBAi(Xil 

CBAt  (XL)  =  —  (1  —  e  ~ri'x0,  when  X,  >  Xj,min;  or  CBAfX,)  =  0,  when  X,  <  Xhmm 

ri 

Where  m,  is  the  marginal  rate  of  dollars  per  unit  attribute,  r,  is  the  (optional)  diminishing  return 
rate,  and  Xmm  is  the  minimum  acceptable  level  (or  zero  point)  for  bigger  is  better  functions.  When 
there  is  no  diminishing  returns  rate,  the  CBA  function  is  simply  a  linear  function  of  (i.e.,  Y  =  m,Xi.) 


124  Mishan  EJ.  Cost-Benefit  Analysis.  5th  ed.,  New  York:  Routledge;  2007. 
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Higher  is  Betters 


Lower  is  Better  => 


Figure  40.  Attribute  monetization  functions  for  the  CBA  value  model. 


Figure  41.  CBA  benefit  vs.  cost  tradespace  (with  Pareto  efficient  set  indicated) 

Each  of  the  Space  Tug  design  alternatives  were  then  evaluated  in  terms  of  the  CBA  benefit  and 
cost,  and  are  plotted  in  Figure  41.  Additionally,  the  Pareto  efficient  set  of  designs  are  indicated 
with  red  triangles  (flat  side  on  left).  Due  to  the  nature  of  CBA  value  no  design  alternatives  are 
rejected,  so  the  full  tradespace  appears  feasible  (N=384).  The  designs  in  the  Pareto  set  tend  to 
have  small  payloads  and  never  have  electric  propulsion  (type  3). 

Measure  of  Effectiveness  (MOE) 

Delta  V  was  used  as  a  single  dimension  Measure  of  Effectiveness  (OAS  2008)125  since  it  represents 
the  fundamental  capability  for  transferring  target  vehicles  from  one  orbital  slot  to  another.  For 
clarity  we  use  a  single  MOE,  but  one  could  use  all  three  attributes,  each  as  a  measure  of 
performance  (MOP)  and  perform  multi-dimensional  Pareto  analysis  to  identify  the  non- 


125  Office  of  Aerospace  Studies.  Analysis  of  Alternatives  (AoA)  Handbook:  A  Practical  Guide  to  Analyses  of 
Alternatives.  AFMC  OAS/A9,  www.oas.kirtland.af.mil,  July  2008. 
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dominated  solutions.  Using  a  performance  metric  as  the  MOE  might  be  considered  equivalent 
to  "not  having  a  value  model."  However,  a  value  model  is  always  being  used  when  a  study  is 
synthesizing  information  to  form  the  basis  of  a  decision,  even  if  a  decision  maker  does  not 
explicitly  acknowledge  a  value  model  as  such. 


Figure  42.  MOE  (Delta  V)  benefit  vs.  cost  tradespace  (with  Pareto  efficient  set  indicated). 

Each  of  the  Space  Tug  design  alternatives  were  evaluated  in  terms  of  the  MOE  benefit  and  cost 
and  are  plotted  in  Figure  42.  Additionally,  the  Pareto  efficient  set  of  designs  are  indicated  with 
cyan  triangles  (flat  side  on  top).  Due  to  the  nature  of  MOE  value,  no  design  alternatives  are 
rejected,  so  the  full  tradespace  appears  feasible  (N=384).  The  designs  in  the  Pareto  set  tend  to 
have  electric  propulsion  since  this  will  result  in  the  largest  delta  V  for  a  given  mass  spacecraft.  All 
of  the  designs  also  have  the  minimum  size  payload,  which  again  reduces  the  overall  dry  mass  of 
the  spacecraft,  resulting  in  additional  delta  V  capability  for  the  Space  Tug  to  impart  on  target 
spacecraft. 


Results 

Now  that  each  of  the  Space  Tug  designs  have  been  evaluated  with  each  of  the  value  models  and 
each  suggests  a  particular  set  of  value  efficient  designs,  the  next  step  is  to  compare  Pareto  sets 
across  the  four  value  models. 

Comparisons  via  Pareto  Sets 

Figure  43  illustrates  the  four  perceived  benefits  versus  costs  tradespaces  across  the  four  value 
models,  with  all  four  Pareto  sets  indicated.  Upon  inspection,  it  appears  that  no  single  point 
appears  in  all  four  Pareto  sets,  but  there  are  a  few  designs  that  appear  in  three  out  of  four  of  the 
sets. 
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Figure  43.  Comparison  of  four  value  tradespaces. 

The  next  step  in  the  study  is  a  more  formal  joint  Pareto  set  analysis  to  determine  the  specifics  of 
apparently  attractive  designs.  This  type  of  analysis  uses  standard  multi-objective  optimization 
techniques  along  with  set  theory  and  has  been  implemented  within  the  IVTea  Suite  (MATLAB®- 
based)  software  mentioned  earlier. 

Joint  Pareto  Analysis 

The  joint  Pareto  analysis  entails  determining  the  Pareto  set  for  each  of  the  four  pairs  of  objectives 
(i.e.,  benefit  and  cost  functions  for  each  of  the  four  value  models).  The  number  of  valid  designs, 
along  with  each  Pareto  set  size  (indicated  as  "0%  PARETO")  is  indicated  in  Figure  44.  It  is 
important  to  notice  that  there  are  zero  "joint"  designs.  Here,  "joint"  means  that  the  design 
appears  in  all  individual  Pareto  sets.  Instead,  there  are  six  "compromise"  designs,  which  are 
determined  by  calculating  the  Pareto  set  across  the  union  of  all  objective  functions.  These 
represent  efficient  solutions  that  are  non-dominated  across  the  full  set  of  objectives. 
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Figure  44.  Joint  Pareto  analysis  with  (a)  four  objective  sets  of  two  objectives  each;  (b)  analysis  results;  (c)  list  of 

six  compromise  designs. 


Upon  closer  inspection,  we  find  that  there  are  also  six  designs  that  are  in  three  out  of  four  Pareto 
sets.  These  are  listed  in  Table  2,  but  two  of  the  six  are  invalid  for  the  MAU  value  model  (meaning 
they  do  not  provide  minimum  acceptable  benefit  in  one  or  more  attributes).  These  designed  are 
considered  "promising"  if  efficiency  across  three  out  of  four  value  models  is  sufficient. 

Table  2.  Promising  designs  that  are  joint  Pareto  efficient  across  three  out  of  four  value  models. 


ID  Number 

Pareto  Efficient  For 

Invalid  For 

1 

2,  3,4 

1 

11 

2,  3,4 

1 

63 

1,  2,3 

95 

1,2,3 

127 

1,2,3 

128 

1,  2,3 

The  details  of  the  promising  designs  are  described  in  Figure  45.  If  we  do  not  consider  designs  1 
and  11,  which  are  invalid  for  the  MAU  value  model,  we  see  a  few  common  design  choices  among 
the  remainder  of  the  designs:  they  all  use  nuclear  propulsion  (type  4),  and  a  large  amount  of  fuel. 
Each  of  these  four  designs  are  highly  attractive  across  the  value  models,  and  are  most  benefit- 
cost  efficient  for  three  out  of  four.  These  are,  however,  very  expensive  systems  (as  determined 
by  the  nuclear  propulsion  and  large  amount  of  fuel).  Finding  less  expensive  alternatives  that  are 
also  robust  to  value  model  choice  would  be  attractive  at  this  point. 

One  other  technique  we  can  leverage  in  trying  to  find  "robust"  solutions  that  are  insensitive  to 
value  model  choice  is  to  calculate  fuzzy  Pareto  efficient  sets  (Fitzgerald  and  Ross  2012)126.  We 
varied  the  fuzziness  level  and  found  that  a  single  design  does  appear  to  be  fully  joint  Pareto 
efficient  at  a  fuzzy  level  of  7%.  This  means  the  design  is  within  7%  of  Pareto  efficiency  for  all  four 
value  models.  An  additional  attractive  feature  of  this  fuzzy  Pareto  design  is  its  lower  cost. 


126  Fitzgerald  ME,  Ross  AM.  Mitigating  contextual  uncertainties  with  valuable  changeability  analysis  in  the  multi¬ 
epoch  domain.  6th  Annual  IEEE  Systems  Conference.  Vancouver,  Canada,  March  2012. 
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63 

95 

127 

128 

S(Cost) 

96.876 

105.24 

900 

1540 

2180 

3020 

S(Cost) 

96.876 

105.24 

900 

1540 

2180 

3020 

S(Cost) 

96.876 

105.24 

900 

1540 

2180 

3020 

S(Cost) 

96.876 

105.24 

900 

1540 

2180 

3020 

S(Cost) 

96.876 

105.24 

900 

1540 

2180 

3020 

dv(DesignforChange) 

0 

0 

0 

0 

0 

0 

dv(DesignD) 

1 

11 

63 

95 

127 

128 

dv(PayloadMass) 

300 

300 

1000 

3000 

5000 

5000 

dv(PropMass) 

30 

300 

10000 

10000 

10000 

30000 

dv(PropType) 

1 

2 

4 

4 

4 

4 

x(Capability) 

300 

300 

1000 

3000 

5000 

5000 

x(DeltaV) 

142.608 

1697.46 

16149.6 

10984.1 

8387.01 

14948.9 

x(ResponseTime) 

1 

1 

1 

1 

1 

1 

x(DeltaV) 

142.608 

1697.46 

16149.6 

10984.1 

8387.01 

14948.9 

iv(BaseMass) 

0 

0 

1000 

1000 

1000 

1000 

iv(DryMass) 

603.6 

639 

5000 

9000 

13000 

17000 

iv(Specificlmpulse) 

300 

450 

1500 

1500 

1500 

1500 

iv(MassFrac) 

0.12 

0.13 

0.2 

0.2 

0.2 

0.2 

iv(WetMass) 

633.6 

939 

15000 

19000 

23000 

47000 

MAU(DM-mau,  Military  All-purpose) 

NaN 

NaN 

0.5692 

0.67607 

0.68376 

0.95796 

u(DM-mau,  x(Capability)) 

NaN 

NaN 

0 

0.7 

1 

1 

u(DM-mau,  x(DeltaV)) 

NaN 

NaN 

0.92299 

0.49017 

0.20941 

0.89489 

u(DM-mau,  x(ResponseTime)) 

1 

1 

1 

1 

1 

1 

AHPbenefit(DM-ahp,  AHP  Military  All-purpose) 

0.20114 

0.2161 

0.4147 

0.53522 

0.68045 

0.74358 

CBAbenefit(DM-cba,  CBA  Military  All-purpose) 

4500 

4702.108 

7075.5553 

7745.3006 

7827.9318 

8517.5532 

Figure  45.  Details  on  the  "promising"  designs. 
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Figure  46.  Comparison  of  benefit  versus  cost  tradespaces  with  compromise,  promising,  and  fuzzy  joint  designs 

indicated. 


Figure  46  illustrates  the  four  tradespaces,  with  the  compromise,  promising,  and  7%  fuzzy  joint 
designs.  Design  52  is  the  7%  fuzzy  joint  Pareto  design  and  represents  the  most  robust  choice  if 
the  decision  maker  is  unsure  of  which  value  model  best  captures  his  preferences.  Interestingly 
this  design  uses  electric  propulsion,  which  was  a  design  choice  absent  from  the  AHP  and  CBA 
Pareto  sets.  Appealingly,  this  design  is  in  the  low  cost  region  of  the  tradespaces.  The  joint  Pareto 
analysis  identified  designs  that  are  most  efficient  across  3  out  of  4  value  models  (tending  to  high 
performance,  high  cost  solutions),  as  well  as  balanced  efficiency  across  all  4  value  models  (lower 
performance,  lower  cost  solution).  Ultimately  the  foregoing  value  model  trade  analysis  doesn't 
prescribe  the  "best"  solution,  but  rather  highlights  several  key  points:  1)  the  choice  of  value 
model  matters  since  it  determines  the  attractiveness  of  each  solution;  2)  each  value  model  will 
likely  highlight  different  systems;  3)  it  is  possible  to  identify  systems  that  do  well  across  multiple 
value  models;  4)  this  type  of  analysis  is  useful  if  the  most  appropriate  value  model  to  use  is 
uncertain  or  likely  to  change.  One  could  theoretically  wrap  an  optimizer  around  the  joint  Pareto 
analysis  to  identify  a  "best"  solution;  however,  this  would  obfuscate  the  pedagogical  aim  of  this 
study. 
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Discussion 


Much  of  the  modeling  literature  tends  to  focus  on  model  formulation  and  validation  in  pursuit  of 
finding  "best"  solutions  (e.g.,  optimization-based  approaches)  (Rhodes  and  Ross  2014)127.  Model 
types  include  performance,  cost,  as  well  as  value  models.  As  pointed  out  earlier  (Ricci  et  al. 
2014)128,  there  is  an  asymmetry  when  validating  performance  models  as  opposed  to  value 
models.  The  former  could  have  ground  truth  as  a  basis  for  validation,  while  the  latter  attempts 
to  put  structure  on  something  that  may  be  fundamentally  subjective  (i.e.,  human  interpretation). 
As  model-centric  methods  proliferate,  and  the  pursuit  of  robust  and  resilient  solutions  becomes 
strategically  important,  analysts  and  engineers  need  to  explore  more  than  just  the  accuracy  and 
sensitivity  of  their  model  results;  they  must  also  explore  the  impact  of  model  choice  itself.  Where 
ground  truth  might  be  available,  model  validation  is  possible,  and  the  impact  of  model  choice 
may  be  interpreted  as  error  introduced  into  the  data.  Where  ground  truth  may  be  unavailable, 
as  may  be  the  case  for  value  models,  understanding  the  impact  of  model  choice  on  data  could 
become  an  essential  part  of  studies.  The  demonstration  case  for  value  model  trading  was 
intended  to  help  identify  key  tasks  and  supporting  infrastructure  for  value  model  trading 
capabilities.  The  case  did  result  in  the  ability  to  use  different  value  model  formulations  on  a 
common  data  set.  The  next  phase  of  the  research  will  continue  analyzing  value  model  trades  in 
this  case,  and  will  develop  a  more  complete  framework  and  process  for  conducting  value  model 
trades  in  general. 


Next  Steps 

The  research  team  will  build  on  the  Phase  2  work  on  value  model  trades  to  further  evolve  the 
framework  and  process.  In  Phase  3,  the  research  team  will  build  on  prior  phase  results  to  further 
evolve  the  framework  and  process  for  conducting  value  model  choice  and  tradeoffs  and  apply 
this  through  an  expanded  case  application  set,  to  validate  the  framework  and  identify  workflow 
considerations.  The  model  choice  and  tradeoff  framework  will  be  expanded  including 
demonstration  cases  beyond  value  models  (to  include  trading  of  other  types  of  models  including 
performance  and  cost  models).  The  expanded  framework  will  consider  alternative  use  cases  for 
the  impact  of  model  choice  and  tradeoffs  on  decision-making.  For  example,  this  includes  the 
context  of  multi-stakeholder  negotiations  using  tradespace  exploration,  where  the  data  source(s) 
(i.e.  "models")  strongly  impact  the  trust  and  framing  of  the  shared  decision  problem 


127  Rhodes  DH,  Ross  AM.  Interactive  Model-Centric  Systems  Engineering  (IMCSE)  Phase  1  Technical  Report.  SERC- 
2014-TR-048-1;  September  2014. 

128  Ricci  N,  Schaffner  MA,  Ross  AM,  Rhodes  DH,  Fitzgerald  ME.  Exploring  stakeholder  value  models  via  interactive 
visualization.  CSER14.  March  2014. 
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Supporting  MPTs 


During  research  activities  within  IMCSE,  a  number  of  opportunities  to  develop  supporting 
methods,  processes,  and  tools  have  arisen.  In  addition  to  the  three  specific  projects  within  the 
three  thrusts  of  the  IMCSE  program,  these  MPTs  will  contribute  to  the  IMCSE  body  of  knowledge 
and  facilitate  knowledge  transfer  to  practice. 


IVTea  Suite 

During  this  phase,  work  on  IVTea  Suite  (Interactive  Value-Driven  and  Tradespace  Exploration  and 
Analysis  Suite)  was  deferred  in  order  to  focus  on  developing  demonstration  prototypes  for  IEEA, 
as  described  in  earlier  sections.  It  is  envisioned  that  new  MPTs  will  emerge  from  the  IEEA 
prototype  implementation  work  during  Phase  3  and  beyond.  These  may  be  compatible  with 
IVTea  Suite,  or  constitute  a  new  complementary  MPT,  focusing  on  facilitating  aspects  of  IMCSE 
research  and  help  with  transition  to  practice. 


Next  Steps 

Going  forward,  IVTea  Suite  will  undergo  refinement  of  user  interface,  data  handling,  as  well  as 
development  of  additional  widgets  that  support  ongoing  research,  as  research  resources  allow. 
Further  development  of  demonstration  prototype  standalone  and  integrated  IEEA  MPTs  will  also 
occur  during  the  next  phase. 
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Interactive  Schedule  Reduction  Model 


The  applications  thrust  in  this  phase  focused  on  the  Interactive  Schedule  Reduction  Model 
project  (ISRM). 

Leveraging  prior  work  from  DARPA  META,  the  Schedule  Reduction  Model  was  extended  with 
interactivity  as  a  central  aspect,  promoting  sensitivity  analyses  and  benchmarking  to  be  the 
central  use  case.  This  report  describes  progress  on  the  ISRM  completed  as  of  the  end  of  Phase  2 
of  the  IMCSE  project129. 


Introduction 

Large  engineering  projects  face  continued  risk  of  significant  cost  and  schedule  overruns  despite 
advances  in  technology  and  management  processes.  Industries  involving  aerospace  and  defense 
systems  are  particularly  afflicted.  A  GAO  report130  highlights  74  instances  of  cost  breaches  in  47 
of  134  major  defense  acquisition  programs  since  1997.  The  largest  factors  responsible  for  unit 
cost  growth  include  engineering  and  design  issues,  schedule  issues,  and  quantity  changes.  Nearly 
40  percent  of  cost  breaches  occurred  after  finalizing  production  decisions,  further  constraining 
options  for  project  restructuring.  A  GAO  report  calls131  for  early  and  continued  systems 
engineering  analysis  aim  to  identify  and  intervene  before  significant  overruns  occur.  Increased 
effort  to  consider  design  alternatives  and  evaluate  achievability  of  objectives  during  design 
reviews  ensure  the  project  meets  requirements  with  available  resources. 

Earth  and  space  science  missions  share  similar  features.  A  NRC  report132  of  NASA  missions  shows 
average  cost  and  schedule  growth  exceeds  20  percent  and  13  of  40  recent  missions  experienced 
excessive  cost  growth.  Commonly  identified  factors  contributing  to  cost  and  schedule  growth 
include  optimistic  and  unrealistic  estimates,  project  and  funding  instability,  instrument  and 
technology  development  problems,  and  launch  service  issues.  Other  contributing  factors  include 
cost  growth  induced  by  schedule  growth  due  to  staffing  and  cost  growth  due  induced  by  cost 
growth  in  another  project  due  to  re-planning.  Most  cost  growth  accumulates  from  development 
issues  after  critical  design  review  (CDR),  even  though  CDR  is  intended  to  mark  the  final  stage  of 
design. 


129  Portions  of  this  report  also  appear  in:  Grogan,  P.T.,  O.L.  de  Week,  A.M.  Ross,  and  D.H.  Rhodes,  "Interactive  models 
as  a  system  design  tool:  Applications  to  system  project  management,"  Proceedings  of  the  2015  Conference  of 
Systems  Engineering  Research,  Hoboken  New  Jersey,  March  2015 

130  U.S.  Government  Accountability  Office  (GAO).  Trends  in  Nunn-McCurdy  Breaches  for  Major  Defense  Acquisition 
Programs.  GAO-11-295R.  Washington,  D.C;  March  2011. 

131  U.S.  Government  Accountability  Office  (GAO).  DOD  Cost  Overruns:  Trends  in  Nunn-McCurdy  Breaches  and  Tools 
to  Manage  Weapon  Systems  Acquisition  Costs.  GAO-11-499T.  Washington,  D.C.;  March  2011. 

132  National  Research  Council  (NRC).  Controlling  Cost  Growth  of  NASA  Earth  and  Space  Science  Missions.  Washington, 
D.C.:  The  National  Academies  Press;  2010 
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The  META  II  Complex  Systems  Design  and  Analysis  (CODA)  project  (Murray  et  al.  2011)133 
investigated  new  design  techniques  relying  on  engineering  software  models  for  early  design 
activities  without  physical  testing.  Key  components  of  the  META  design  process  include 
deliberate  use  of  layers  of  abstraction,  development  and  use  of  a  component  model  library 
(C2M2L),  and  virtual  verification  and  validation  processes.  Past  work  (de  Week  2012)134 
developed  the  Design  Flow  Model  (DFM)  as  a  prototype  system  dynamics  (SD)  tool  to  evaluate 
the  feasibility  of  a  five-fold  speedup  in  system  development  under  the  META-enabled  process, 
showing  potential  for  a  five-fold  speedup  for  projects. 

The  IMCSE  program  builds  on  these  ideas  to  create,  validate,  and  transition  methods,  processes, 
and  tools  to  rapidly  model  the  critical  aspects  of  systems,  especially  those  that  facilitate 
collaborative  system  development.  IMCSE  aims  to  develop  transformative  results  in  engineering 
projects  through  intense  human-model  interaction.  The  Interactive  Schedule  Reduction  Model 
(ISRM)  is  one  of  three  activities  in  the  first  and  second  phases  of  IMCSE  to  demonstrate  web- 
based  technologies  as  new  methods  for  human-model  interaction  enabling  rapid  sensitivity 
analysis  of  various  factors.  It  uses  the  DFM  as  a  use  case  to  explore  alternative  systems 
development  processes  and  resource  allocations  and  determine  their  potential  impact  on 
program  schedule. 


Background  and  Objectives 


The  Role  of  Design  Tools 

As  discussed  above,  aerospace  and  defense  projects  are  particularly  afflicted  by  cost  and 
schedule  overruns.  A  variation  of  this  pattern  has  been  popularized  by  Augustine's  Law  XVI135 
which  observes  that  aircraft  unit  costs  increase  exponentially  while  budgets  increase  linearly, 
leading  to  the  seemingly-absurd  case  where  the  entire  defense  budget  affords  just  one  aircraft 
by  2054.  Investigating  the  source  of  cost  growth  in  fixed-wing  aircraft,  a  RAND  study  (Arena  et  al. 
2010)136  estimates  economy-driven  factors  contribute  only  about  a  third  of  cost  growth. 
Customer-driven  factors  attribute  the  remaining  two-thirds  with  major  contributions  from 
complexity  of  performance  characteristics  and  airframe  material. 


133  Murray  B,  Pinto  A,  Skelding  R,  de  Week  0,  Zhu  H,  Nair  S,  Shougarian  N,  Sinha  K,  Bopardikar  S,  Zeidner  L.  META  II 
Complex  Systems  Design  and  Analysis  (CODA)  Final  Report.  AFRL-RZ-WP-TR-2011-2102.  Wright-Patterson  Air 
Force  Base,  Ohio:  Air  Force  Research  Laboratory;  August  2011. 

134  de  Week  OL.  "Feasibility  of  a  5X  Speedup  in  System  Development  Due  to  META  Design."  DETC2012-70791. 
International  Design  Engineering  Technical  Conferences  &  Computers  and  Information  in  Engineering  Conference. 
Chicago,  Illinois:  August  2012. 

135  Augustine  NR.  Augustine's  Laws.  Sixth  Edition.  Reston,  Virginia:  AIAA;  1997. 

136  Arena  MV,  Younossi  0,  Brancato  K,  Blickstein  I,  Grammich  CA.  Why  Has  the  Cost  of  Fixed-Wing  Aircraft  Risen? 
MG696-1.2.  The  RAND  Corporation.  Santa  Monica,  California;  2010. 
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There  are  many  descriptions  and  definitions  of  complexity  in  literature;  however,  a  unifying 
perspective  (Suh  1999)137  for  system  design  relates  it  to  uncertainty  in  meeting  functional 
requirements  (FRs)  within  cost  and  schedule  constraints.  Sources  of  complexity  (Rhodes  and  Ross 
2008)138  include  structural  (components  and  interrelationships),  behavioral  (functional  response 
to  inputs),  contextual  (outside  circumstances),  temporal  (time  dynamics),  and  perceptual 
(stakeholder  preferences)  factors.  Most  efforts  to  quantify  complexity  focus  on  structural 
features  where  information-  or  entropy-based  methods  (Sinha  and  de  Week  2012)139  define  a 
complexity  metric  as  a  function  of  system  components,  their  interconnections,  and  overall 
architecture.  Application-specific  studies  (Deshmukh  et  al.  1998)140  (Frizelle  and 
Woodcockl995)141  show  systems  with  higher  complexity  measures  can  provide  higher  levels  of 
performance  than  simpler  systems  if  they  are  optimally  managed. 

Downsides  of  complexity  arise  from  limitations  in  individual  and  social  cognition.  To  emphasize 
this  distinction,  consider  descriptive  and  perceived  dimensions  (Schlindwein  and  Ison  1994)142. 
Descriptive  complexity  is  the  objective  system  property  related  to  information  content  as 
described  in  entropy-based  measures.  Perceived  complexity  is  the  subjective  property  related  to 
uncertainty  in  meeting  FRs  due  to  an  observer's  incomplete  knowledge  of  required  information. 
This  project  assumes  perceived  and  descriptive  complexity  are  correlated,  and  constitute  a 
tradeoff  between  efficiency  and  robustness  generally  observed143  in  systems  architecting  (Doyle 
and  Csete  2011).  Descriptive  complexity  can  improve  efficiency  of  meeting  a  given  set  of  FRs 
under  expected  (nominal)  conditions  while  perceived  complexity  reduces  robustness  by  adding 
uncertainty  to  achieving  FRs  within  cost  and  schedule  constraints.  Due  to  perceptual  limitations, 
seemingly-efficient  designs  may  realize  poor  performance  and  produce  "robust-yet-fragile" 
conditions(Alderson  and  Doyle  2010)144. 

The  notional  tradespace  of  system  architectures  in  the  figure  below  illustrates  the  efficiency- 
robustness  relationship.  The  ideal  design  (upper  right)  is  limited  by  constraints  on  descriptive  and 


137  Suh  NP.  "A  Theory  of  Complexity,  Periodicity  and  the  Design  Axioms."  Research  in  Engineering  Design  1999; 
11(2):116-131. 

138  Rhodes  DH,  Ross  AM.  "Five  Aspects  of  Engineering  Complex  Systems:  Emerging  Constructs  and  Methods."  4th 
Annual  IEEE  Systems  Conference.  San  Diego,  California:  April  2008. 

139  Sinha  K,  de  Week  OL.  "Structural  Complexity  Metric  for  Engineering  Complex  Systems  and  its  Application."  14th 
International  DSM  Conference.  Kyoto,  Japan:  September  2012. 

140  Deshmukh  AV,  Talavage  JJ,  Barash  MM.  "Complexity  in  Manufacturing  Systems  Part  1:  Analysis  of  Static 
Complexity."  HE  Transactions  1998;  30(7):645-655. 

141  Frizelle  G,  Woodcock  E.  "Measuring  Complexity  as  an  Aid  to  Developing  Operational  Strategy."  International 
Journal  of  Operations  &  Production  Management  1995;  15(5):26-39. 

142  Schlindwein  SL,  and  Ison  R.  "Fluman  Knowing  and  Perceived  Complexity:  Implications  for  Systems  Practice." 
Emergence:  Complexity  and  Organization  2004;  6(3):27-32. 

143  Doyle  JC,  Csete  M.  "Architecture,  Constraints,  and  Behavior."  Proceedings  of  the  National  Academy  of  Sciences 
of  the  United  States  of  America  2011;  108(3):15624-15630. 

144  Alderson  DL,  Doyle  JC.  "Contrasting  Views  of  Complexity  and  Their  Implications  for  Network-Centric 
Infrastructures."  IEEE  Transactions  on  Systems,  Man,  and  Cybernetics  -  Part  A:  Systems  and  Flumans  2010; 
40(4):839-852. 
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perceived  complexity.  Robust  designs  (upper  left)  tend  to  be  inefficient  due  to  constraints  on 
descriptive  complexity  required  to  anticipate  responses  to  uncertainties  (e.g.  consider  the  KISS 
principle:  "keep  it  simple  stupid"  reportedly  coined  by  Kelly  Johnson).  Efficient  solutions  (lower 
right)  tend  to  be  fragile  due  the  inability  to  anticipate  responses  to  uncertainties  caused  by  high 
perceived  complexity. 

Design  studies  (Hershi  and  Frey  2002), 145  (Singha  2014), 146  (Flagler  et  al.  2014), 147  (Grogan 
2014)148  consistently  show  a  super-linear  relationship  between  objective  complexity  measures 
and  effort  to  complete  a  design  with  fixed  requirements.  Although  perceived  complexity  cannot 
be  observed  as  a  hidden  intermediate  variable,  this  project  hypothesizes  it  to  be  a  contributing 
mechanism  for  cost  and  schedule  overruns.  Consider  the  illustrative  example  in  the  figure  below 
.  A  new  project  seeks  to  increase  performance  over  past  projects  with  an  increase  in  descriptive 
complexity  (a).  Perceived  complexity  is  assumed  to  be  related  to  descriptive  complexity  by  a 
monotonically-increasing  function  dependent  on  the  system  and  its  observers  (b).  Differences  in 
function  slope  and  shape,  for  example,  distinguish  between  VLSI  and  mechanical  design  (Whitney 
1996)149.  Project  cost  and  schedule  is  a  super-linear  function  of  perceived  complexity  (c)  which 
assigns  higher  cost  and  schedule  to  deal  with  high  perceived  complexity. 


Efficiency 
Limited  by  Low 
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Complexity 


Robustness 
Limited  by  High 
Perceived  * 
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Nominal  Efficiency 


Efficiency  versus  robustness 
as  an  architectural  trade  in 
design. 


Hypothesized  mechanisms  of  cost  and  schedule  overruns:  new  projects 
increase  descriptive  complexity  to  meet  performance  goals  (a),  which 
increases  perceived  complexity  (b),  and  leads  to  cost  and  schedule  overruns 
of  linear  extrapolation  from  past  projects  (c). 


This  theoretical  model  highlights  three  potential  sources  of  cost  or  schedule  estimation  errors: 
1)  errors  in  the  level  of  descriptive  complexity  to  meet  target  performance,  2)  errors  relating 
descriptive  and  perceived  complexity,  and  3)  errors  relating  perceived  complexity  and  cost  and 
schedule.  Errors  in  (3)  are  particularly  biased  towards  cost  and  schedule  under-estimation. 
Humans  have  difficulty  in  estimating  geometric  or  exponential  growth,  instead  using  linear 


145  Hirschi  NW,  Frey  DD.  "Cognition  and  Complexity:  An  Experiment  on  the  Effect  of  Coupling  in  Parameter  Design." 
Research  in  Engineering  Design  2002;  13(3):123-131. 

146  Sinha  K.  Structural  Complexity  and  its  Implications  for  Design  of  Cyber-Physical  Systems.  PhD  thesis.  Cambridge, 
Massachusetts:  Massachusetts  Institute  of  Technology;  2014. 

147  Flager  F,  Gerver  DJ,  Kallman  B.  "Measuring  the  Impact  of  Scale  and  Coupling  on  Solution  Quality  for  Building 
Design  Problems."  Design  Studies  2014;  35(2):180-199. 

148  Grogan  PT.  Interoperable  Simulation  Gaming  for  Strategic  Infrastructure  Systems  Design,  PhD  thesis.  Cambridge, 
Massachusetts:  Massachusetts  Institute  of  Technology,  2014. 

149  Whitney  DE.  "Why  Mechanical  Design  Cannot  be  like  VLSI  Design."  Research  in  Engineering  Design  1996;  8(3):125- 
138. 
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extrapolations  in  intuitive  assessment  (Stango  and  Zinman  1995)150.  Linearizing  results  of  past 
projects,  as  shown  above,  (c)  leads  to  under-estimations  characteristic  of  large  or  complex 
projects  beyond  existing  experience. 


There  are  two  approaches  to  address  cost  and  schedule  growth  in  engineering  projects:  pursue 
more  conservative  designs  with  lower  descriptive  complexity  (at  the  cost  of  lower  performance) 
or  improve  the  designers'  perception.  This  project  seeks  improved  perception  to  achieve  desired 
performance  of  descriptively-complex  systems  at  lower  cost,  expanding  the  space  of  feasible 
designs  in  the  figure  below  (left). 
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Improved  perception  enables  new  designs 
outside  the  previously-feasible  region. 


.  Proposed  role  of  design  methods  and  tools:  new  tools  reduce 
perceived  complexity  (a)  leading  to  lower  cost  and  schedule  (b). 


Design  methods  and  tools  are  proposed  to  reduce  perceived  complexity  and  help  designers 
acquire  knowledge  to  manage  descriptively-complex  systems.  Methods  such  as  filtering, 
abstraction  or  generalization,  and  automation  reduce  a  problem  to  its  essential  features  and 
apply  pre-defined  procedures  at  lower  levels.  Computational  tools  provide  extensive  memory, 
rapid  communication,  and  new  human-computer  interfaces  for  advanced  visualization.  The 
figure  above  (right)  illustrates  the  effect  of  design  tool  innovations  on  the  functional  relationship 
between  descriptive  and  perceived  complexity  (a),  ultimately  reducing  cost  and  schedule  (b).  In 
summary,  tools  improving  designer  perception  are  hypothesized  to  reduce  design  effort  by 
anticipating  product  performance  under  a  wider  range  of  conditions. 


Modeling  Tools  in  Systems  Engineering 

Recent  SE  practices  show  increased  focus  on  model-centric  tools  to  support  design  activities. 
Model-based  systems  engineering  (MBSE)  is  defined  by  INCOSE  (2007)151  as  a  "formalized 
application  of  modeling  to  support  system  requirements,  design,  analysis,  verification,  and 
validation  activities  beginning  in  the  conceptual  design  phase  and  continuing  throughout 
development  and  later  life  cycle  phases."  MBSE  aims  (Friedenthal  et  al.  20  1  2)152  to  replace  labor- 


150  Stango  V,  Zinman  J.  "Exponential  Growth  Bias  and  Household  Finance."  The  Journal  of  Finance  2009; 
64(6):2807:2849. 

151  International  Council  on  Systems  Engineering  (INCOSE).  Systems  Engineering  Vision  2020  Version  2.03.  TP-2004- 
004-02.  September  2007. 

152  Friedenthal  S,  Moore  A,  Steiner  R.  A  Practical  Guide  to  SysML.  Second  Edition.  Waltham,  Massachusetts:  Elsevier; 
2012. 
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intensive,  error-prone,  and  cumbersome  document-based  processes  with  model-based  methods 
to  improve  specification  and  design  quality,  specification  and  design  reuse,  and  development 
team  communications.  In  addition  to  efficiency  gains  commonly-identified,  evidence  (Sterman 
1994)153  of  active  participation  in  model-building  leading  to  more  effective  learning  may  allow 
MBSE  efforts  to  reduce  perceived  complexity  of  descriptively-complex  systems. 

The  META  II  Complex  Systems  Design  and  Analysis  (CODA)  project  (Murray  et  al.  2011)154 
explored  use  of  model-based  techniques  in  design  activities.  It  developed  three  key  mechanisms 
to  reduce  cost  and  schedule  overruns.  First,  multiple  layers  of  abstraction  allow  concepts  to  be 
quickly  developed  and  assessed  at  a  coarse  level  and  refined  during  detailed  design.  Second, 
designers  develop  and  maintain  a  trusted  component  model  library  (C2M2L)  to  limit  costly 
model-building  and  validation  exercises.  Third,  re-design  cycles  take  place  in  virtual 
environments,  allowing  designers  to  rapidly  evaluate  concepts  and  find  required  changes  sooner. 

Past  work  (de  Week  2012)155  developed  the  Design  Flow  Model  (DFM)  as  a  system  dynamics  (SD) 
model  to  assess  differences  between  traditional  sequential  stage-gate  development  processes 
and  the  flexible  META-enabled  design  methods  for  projects  in  the  Adaptive  Vehicle  Make  (AVM) 
program  portfolio.  The  SD  model  formalism  defines  stock  (accumulations)  and  flow  (rates  of 
change)  variables  as  functions  of  other  model  components.  Numerical  techniques  integrate 
stocks  as  a  system  of  differential  equations  in  a  time-stepped  simulation.  DFM  stocks  are  SE 
activity  products  from  requirements  elicitation,  architectural  exploration,  design  and  integration, 
verification,  and  validation.  DFM  flows  quantify  factors  influencing  work  products  such  as  change 
generation,  time  pressure,  and  efficiency. 

Past  results  of  simulated  projects  show  an  idealistic  project  requires  42.25  months  and  $27. 9M 
of  non-recurring  engineering  (NRE)  cost  to  complete.  When  considering  rework  due  to  change 
generation  (i.e.  problems  arising  from  limits  on  perception),  however,  a  realistic  project  requires 
70  months  and  $51. 9M  in  NRE  costs  (65%  schedule  growth  and  86%  cost  growth).  An  equivalent 
META-enabled  project  with  partial  model  library  completion  requires  only  15.75  months  and 
$31. 5M  in  NRE  costs— a  speedup  factor  of  4.4.  Most  performance  gains  are  due  to  early  design 
work  at  higher  levels  of  abstraction  which  catches  problems  earlier  in  the  development  cycle. 


Platforms  for  Modeling  and  Simulation 

While  initial  results  are  promising,  the  DFM  requires  additional  work  to  evaluate  sensitivity  to 
key  input  parameters  and  determine  its  applicability  beyond  the  AVM  program  portfolio. 


153  Sterman  JD.  "Learning  In  and  About  Complex  Systems."  System  Dynamics  Review  1994:  10(2-3):291-330. 

154  Murray  B,  Pinto  A,  Skelding  R,  de  Week  0,  Zhu  H,  Nair  S,  Shougarian  N,  Sinha  K,  Bopardikar  S,  Zeidner  L.  META  II 
Complex  Systems  Design  and  Analysis  (CODA)  Final  Report.  AFRL-RZ-WP-TR-2011-2102.  Wright-Patterson  Air 
Force  Base,  Ohio:  Air  Force  Research  Laboratory;  August  2011. 

155  de  Week  OL.  "Feasibility  of  a  5X  Speedup  in  System  Development  Due  to  META  Design."  DETC2012-70791. 
International  Design  Engineering  Technical  Conferences  &  Computers  and  Information  in  Engineering  Conference. 
Chicago,  Illinois:  August  2012. 
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Interactive  "what-if"  planning  models  have  been  shown  (Sharon  et  al.  2009)156  to  provide 
benefits  in  similar  project  management  contexts  and  may  be  effective  to  allow  practitioners  to 
understand  and  evaluate  benefits  of  applied  MBSE  efforts  such  as  META.  More  importantly,  the 
DFM  serves  as  a  microcosm  of  the  broader  challenges  to  model-based  design  and  serves  as  a  use 
case  for  new  methods  to  generate  and  analyze  large  data  sets.  Advancing  these  broad  objectives 
revisits  underlying  tools  and  techniques  for  contemporary  modeling. 

The  DFM  was  implemented  in  Vensim157,  an  industry-standard  tool  for  SD  model  development 
and  execution.  Vensim  provides  high-performance  simulation  with  sensitivity  analysis,  data 
import,  and  optimization  capabilities.  Flowever,  it  follows  a  paradigm  where  models  are  usually 
developed  by  one  designer  with  one  formalism  for  use  by  one  individual  to  carry  out  one 
experiment.  Some  Vensim  products  provide  supplementary  features  for  broader  interaction  with 
models  such  as: 

•  Command  scripts:  allow  a  licensed  user  to  automate  model  executions, 

•  Open  Database  Connectivity  (ODBC):  allow  a  licensed  Windows  user  to  read  from  or  write 
to  an  ODBC-supported  database, 

•  Dynamic  Data  Exchange  (DDE):  allow  a  licensed  Windows  user  to  exchange  information 
with  other  DDE-supported  applications,  and 

•  Dynamic  Linked  Library  (DLL):  allow  a  licensed  user  to  integrate  Vensim  functionality  in 
other  applications. 

Licenses  restrict  redistribution  of  Vensim  executables  and  libraries.  While  most  users  can  license 
a  free  Vensim  Model  Reader  to  execute  (but  not  edit)  models,  the  Vensim  tool  cannot  be 
modified  to  integrate  new  capabilities  without  developing  a  separate  application  and  linking  the 
DLL  which  is  separately  licensed.  These  limitations  constrain  the  ability  to  share,  refine,  and 
customize  model-based  tools. 

A  new  modeling  paradigm  (Jacobs  2005)158  emphasizes  collaborative  modeling  among  multiple 
designers  for  multiple  users  and  multiple  applications.  However  limited  progress  has  been 
observed  to  date.  For  example,  a  survey  (Boer  et  al.  2009)159  shows  little  use  of  interoperable 
simulation  standards  outside  defense  applications  due  to  the  complexity  and  cost  of  runtime 
applications  and  incompatibility  with  commercial  packages  common  in  industry.  In  contrast, 
innovations  in  web-  and  browser-based  technologies  in  recent  years  represent  the  most 
advanced  techniques  to  share  and  use  data  and  could  form  the  basis  of  collaborative  modeling. 
ISRM  intends  to  transition  core  features  of  web-based  applications — open-source  core  libraries 


156  Sharon  A,  de  Week  OL,  Dori  D.  "Is  There  a  Complete  Project  Plan?  A  Model-based  Project  Planning  Approach." 
Nineteenth  Annual  International  Symposium  of  the  International  Council  on  Systems  Engineering  (INCOSE). 
Singapore:  2009. 

157  Ventana  Systems  Incorporated.  Vensim  version  6.3.  http://vensim.com,  accessed  22-Sept  2014. 
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and  loosely-coupled  interfaces  based  on  HTTP-based  data  exchange — to  modeling  and 
simulation. 

There  exist  several  SD  modeling  tools  on  web  platforms.  Forio  Simulate160  is  a  commercial  web- 
based  service  addressing  similar  goals  of  this  project;  however  it  is  closed-source  and  proprietary. 
Insight  Maker161  is  a  similar  open  web-based  modeling  tool  but  it  provides  a  graphical  tool  as  a 
stand-alone  modeling  environment  rather  than  a  general-purpose  library.  Lower-level  libraries 
such  as  SIM.JS162  support  discrete  event  simulation  with  features  such  as  random  number 
generation  but  do  not  support  the  SD  formalism.  Other  mathematical  computing  libraries  such 
as  Numeric  Javascript163  and  Sylvester164  implement  vectors  and  matrixes  but  do  not  provide 
integrators  required  for  the  SD  formalism. 


Development  Approach 

Based  on  limitations  of  existing  SD  platforms.  Figure  47  below  outlines  ISRM  objectives  as  six 
tasks  in  two  phases. 
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Figure  47  ISRM  development  approach  in  Phase  1  (a)  and  Phase  2  (b)  with  six  tasks. 


Phase  1  of  this  project  transformed  the  existing  DFM  into  a  browser-based  tool  to  facilitate 
interaction  and  extension.  It  allows  users  to  run  simulation  executions,  view  or  export  numerical 
results,  and  override  input  parameters.  Task  1  develops  an  application  programming  interface 
(API)  to  execute  a  model  and  interpret  results  in  a  browser.  Task  2  ports  the  existing  DFM  from 
Vensim  to  JavaScript  using  the  API.  Task  3  develops  a  user  interface  (UI)  to  allow  interactive 
model  exploration  in  a  browser  environment. 


Phase  2  developed  a  service-based  application  to  compose  and  query  datasets  across  model 
executions.  It  allows  users  to  specify  ranges  of  input  parameters,  aggregate  existing  datasets,  and 
execute  models  to  generate  and  store  new  data.  Task  4  develops  a  service  API  to  collect  and 
query  results  across  model  executions.  Task  5  implements  the  backend  components  to  interact 


160  Forio  Simulate,  http://forio.com/simulate.  accessed  22-Sept  2014. 

161  Insight  Maker,  http://insightmaker.com,  accessed  22-Sept  2014. 

162  SIM.JS  version  0.26.  http://simjs.com,  accessed  22-Sept  2014. 

163  Numeric  Javascript  version  1.2.6.  http://www.numericjs.com,  accessed  22-Sept  2014. 

164  Sylvester  version  0.1.3.  http://sylvester.icoglan.com,  accessed  22-Sept.  2014. 
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with  the  services.  Finally,  Task  6  develops  a  Ul  to  provide  allow  users  to  command  model 
execution  under  conditions  of  interest  and  show  and  interpret  large  quantities  of  information. 


Standalone  ISRMTool 

The  standalone  ISRM  tool  seeks  to  replicate  the  Vensim-based  DFM  in  a  browser  environment. 
This  section  reviews  the  tasks  to  define  a  model  application  programming  interface  (API), 
implement  and  validate  the  model,  and  design  the  user  interface  (Ul). 


JavaScript  Modeling  and  Simulation  (MAS)  API 

The  JavaScript  language  was  not  originally  intended  for  numerical  computation  and  no  existing 
libraries  are  applicable  to  time-evoked  simulation.  This  section  defines  a  JavaScript  Modeling  and 
Simulation  (MAS)  API  for  a  portion  of  the  system  dynamics  (SD)  formalism  shown  as  an  object 
class  diagram  in  Figure  48. 

SD  components  descend  from  a  common  Entity  class  which  establishes  required  attributes 
(unique  id)  and  methods  to  initialize  (init),  and  advance  time  (tick/tock).  The  two-step  time 
advance  avoids  order  dependence  by  pre-computing  (tick)  and  then  committing  (took)  state 
changes.  In  comparison,  Vensim  sequences  the  order  of  equations  to  eliminate  dependence.  The 
tick/tock  method  can  approximate  simultaneous  equations;  however  it  is  currently  limited  to 
single-iteration  numerical  integration  methods  such  as  explicit  (forward)  Euler,  utiis  provides 
utility  functions  such  as  generating  a  globally-unique  identifier  (guid)  and  replicating  the  integer- 
part  method  from  Vensim  (intPart). 


mas. sim. Simulator 


entities:  Array 
handlers:  Array 
initTime:  Number 
maxTime:  Number 
method:  Object 
time:  Number 
timeStcp:  Number 


advanceQ:  Undefined 
entity(id):  Object 
execute():  Undefined 
init():  Undefined 
isComplete():  Boolean 
off(events,  handler):  Undefined 
on(events,  handler):  Undefined 
trigger(event,  data):  Undefined 
value(id):  Number 


mas.sim. Entity 


id:  String 


init(sim):  Undefined 
tick(sim):  Undefined 
tock():  Undefined 


mas. util. Utils 


guid():  String 

intPart( Number):  Number 


mas.sim.ExplicitEuler 


name:  String 


integrated,  dx  dt,  dt):  Number 


mas.sim.LoggingSimulator 

log:  Object 


Figure  48.  Class  diagrams  for  the  JavaScript  API  for  SD  models 


Entity  subclasses  define  components  in  the  SD  formalism.  Parameter  defines  components  with  a 
constant  value.  Flow  defines  components  with  value  dependent  on  other  components, 
functionally  defined  by  overriding  a  method  (getvaiue).  stock  defines  components  with  a  state 
variable  numerically  integrated  during  a  simulation  with  derivative  specified  by  overriding  a 
method  (getDerivative).  Delayl  and  Smooth  define  first-order  exponential  delay  and  smoothing  of 
an  input  signal  specified  by  overriding  a  method  (getinput).  The  sim  argument  provides  access  to 
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the  simulation  context  including  integration  method  (  sim. method,  integrate  (...)),  time  (sim.  time),  Of 
entity  velues  (sim.  value  (...) ). 

Simulator  aggregates  Entity  objects  to  perform  a  time-managed  simulation.  The  initialize  method 
(init)  initializes  all  entities  and  triggers  an  "init"  event.  The  advance  method  (advance)  ticks/tocks 
all  entities,  increments  simulation  time,  triggers  an  "advance"  event,  and  triggers  a  "complete" 
event  if  complete.  The  default  completion  check  method  (iscompiete)  compares  the  current 
simulation  time  against  the  specified  maximum  time  (maxTime).  Finally,  event  handling  methods 
bind  handlers  to  events  (on),  remove  handlers  (off),  and  trigger  events  (trigger).  Similarly, 
Loggings imulator  logs  time-based  attribute  values. 


Model  Implementation 

The  JavaScript  port  of  the  DFM  instantiates  SD  entities  composed  in  a  separate  object  instance. 
The  Model  class  (figure  below  left)  shows  attributes  to  identify  the  model  version  and  override 
parameter  values  and  component  SD  entities.  Each  model  component  includes  additional 
attributes  to  define  semantic  names,  descriptions,  and  units  as  documentation.  For  example,  the 
following  defines  the  NRE  Cost  stock: 

new  mas . sd . Stock ( { 

id:  ' nreCost ' , 
name:  "NRE  Cost", 

desc:  "Non-recurring  engineering  cost.", 
units:  "$", 

getDerivative :  function (sim)  { 

return  sim. value ( "spendRate" ) ; 


} 


This  component  overrides  the  default  getDerivative  method  to  access  the  Spending  Rate  flow 
value. 
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form.  Model 

name:  String 
version:  String 
params:  Object 
entities:  Array 


|  Parameter 


25, 


i  Flow 


10, 


J  Stock 


Delay  1 


\  Smooth 


The  ISRM  model  class  instantiates  required 
simulation  entities  for  the  SD  formalism. 


ISRM  standalone  model  performance  benchmark  under 
four  conditions.  Error  bars  show  95%  confidence  interval 
over  100  trials. 


The  Vensim  DFM  and  JavaScript  ISRM  are  cross-validated  by  comparing  outputs  at  each  time  step 
under  both  the  META  and  no-META  conditions.  Differences  in  numerical  precision  (JavaScript 
uses  double-precision  while  most  versions  of  Vensim  only  use  single-precision)  restrict  identical 
results.  The  no-META  condition  produces  approximately  equal  results  in  both  tools.  The  META 
condition  produces  small  differences  in  intermediate  variables  which  can  appear  as  large  relative 
differences  for  small  quantities.  For  example,  the  JavaScript  model  shows  -0.068  pending  changes 
at  time  1.5  while  the  Vensim  model  shows  -0.038,  a  relative  difference  of  more  than  80%.  Despite 
some  transient  values,  the  relative  difference  in  final  outcomes  between  the  two  tools  is  less 
than  6.9-10  5  for  all  variables. 

A  performance  benchmark  the  figure  above  (right)  evaluates  baseline  execution  time  using 
Google  Chrome  version  39  with  an  Intel  Core  i5-760  CPU.  Test  conditions  vary  META  input 
conditions  and  logging  of  intermediate  values.  Results  range  between  35  and  130  milliseconds 
for  a  120-month  simulation  with  0.25  month  time  steps.  Higher  execution  times  arise  from  longer 
project  durations  without  META  processes  and  from  data  operations  due  to  logging.  Although 
results  cannot  be  compared  to  Vensim  due  to  license  and  application  limitations,  the  small 
magnitude  of  around  10  executions  per  second  provides  a  compelling  case  that  JavaScript-based 
models  are  suitable  for  interactive  interfaces. 
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Figure  49.  Screen  captures  comparing  user  interfaces  for  the  Vensim-based  DFM  (top)  and  browser-based  ISRM 

(bottom). 


107 


Standalone  User  Interface 


The  standalone  ISRM  user  interface  (Ul)  is  a  web  page  structured  and  styled  with  HTML  and  CSS 
and  controlled  with  JavaScript.  Figure  49Error!  Reference  source  not  found,  compares  the 
Vensim  Ul  (top)  to  the  ISRM  Ul  (bottom).  Buttons  on  the  top  section  control  simulations,  the 
middle  section  plots  data,  and  the  bottom  section  visualizes  a  stock-and-flow  diagram.  jQuery165 
handles  form  inputs  and  event  handling,  Flot166  plots  data,  and  kinetic.js167  manages  the  stock- 
and-flow  diagram.  Users  click  and  drag  stocks  (rectangles),  flows  (black  labels),  parameters  (blue 
labels),  and  shadow  variables  (gray  labels)  to  customize  the  display.  Double-clicking  a  field  opens 
a  jQuery  Ul168  dialog  widget  to  edit  parameter  values,  view  flow  values,  view/edit  stock  values, 
toggle  plotting,  and  view  documentation. 


Standalone  Tool  Limitations 

The  biggest  limitation  of  standalone  tool  compared  to  existing  SD  tools  arises  from  a  fixed  model 
structure  in  the  Ul  where  the  user  can  only  change  parameter  values.  A  few  flag-based 
parameters  such  as  META  features  or  Change  Generation  toggle  some  features,  but  all  other 
parameters  such  as  Productivity,  Model  Library  Coverage  and  Integrity,  and  Staff  Efficiency  only 
change  value.  Input  parameters  alone  cannot  modify  the  assumed  model  structure  or  behavior. 

A  number  of  assumptions  in  the  DFM  limit  its  applicability  to  broader  engineering  projects.  For 
example,  it  does  not  enforce  staffing  level  constraints  for  design  processes  and  assumes  a  ramp- 
up  profile  for  initial  requirements  elicitation,  implications  of  complexity  for  design  productivity, 
and  mechanics  of  change  generation.  Changingthese  assumptions  requires  a  new  model-building 
activity  rather  than  the  current  model-using  activity.  While  the  JavaScript  API  is  particularly 
amenable  to  overriding  existing  definitions,  the  ISRM  Ul  requires  hard  coding.  Adding  model¬ 
building  activities  to  the  ISRM  will  require  a  significant  development  effort  to  validate  functional 
behaviors  and  automate  layout  of  the  stock  and  flow  diagram. 

Additionally,  the  tick/tock  time  advancement  method  in  MAS  restricts  numerical  integration  to 
one-step  methods  such  as  explicit  (forward)  Euler.  More  precise  methods  such  as  fourth  order 
Runge-Kutta  (RK4)  require  either  a  centralized  state  update  procedure  or  more  iterative  periods 
to  estimate  intermediate  values.  Future  work  should  adapt  the  tick/tock  procedure  to  allow  for 
other  numerical  integration  or  simulation  assumptions.  Future  work  may  also  extend  MAS  to 
consider  other  SD  functions  beyond  Delayl  and  Smooth. 


165  jQuery  version  2.0.3.  http://iquery.com,  accessed  22-Sept  2014. 

166  Flot  version  0.8.1.  http://flotcharts.org,  accessed  22-Sept  2014. 

167  Kinetic.js  version  5.1.0.  http://www.kineticjs.com,  accessed  22-Sept  2014. 

168  jQuery  Ul  version  1.10.3.  http://jqueryui.com,  accessed  22-Sept  2014. 
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Service-based  ISRM  Application 


The  service-based  ISRM  application  seeks  to  extend  the  capabilities  of  the  standalone  tool  to 
compose,  query,  and  visualize  data  across  multiple  model  executions.  This  section  reviews  the 
tasks  to  define  a  service  application  programming  interface  (API),  implement  and  validate  the 
backend  components,  and  design  the  user  interface  (Ul). 


Services  API 

The  ISRM  services  API  defines  an  interface  to  local  and  remote  model  execution  and  individual 
and  aggregated  data  queries.  Service  requests  and  responses  rely  on  a  common  JavaScript  object 
notation  (JSON)  data  format  shown  below  (left).  A  Result  object  includes  model  information, 
simulation  settings,  input  parameters,  time-stepped  outputs  and  final  values,  and  a  user-defined 
tag. 

The  table  below  (right)  lists  three  services  and  corresponding  routing  URLs.  The  result  service 
allows  a  user  to  query  a  particular  result  or  submit  new  data  from  a  local  model  execution.  The 
execute  service  allows  a  user  to  submit  a  request  for  remote  model  execution.  Finally,  the  data 
service  allows  a  user  to  query  aggregated  data  from  the  complete  set  of  results.  Aggregated 
queries  overcome  minimum  transaction  times  compared  to  a  large  number  of  individual  queries. 

ISRM  data  and  execution  services. 


ISRM  service-based  data  model  used  to  structure  and 
query  aggregated  data,  individual  result,  and  remote 
execution  services. 


Method  and  URL 

Action 

GET  /results/:id 

Queries  a  result  by  ID. 

POST  /results 

Submits  a  new  local  result 
contained  in  the  in  request 
body.  Responds  with  its 
assigned  ID. 

POST  /execute 

Submits  a  remote  model 

execution  defined  in  the 
request  body.  Responds  with 
its  assigned  ID. 

GET /data 

Queries  an  aggregated  list  of 
all  data  objects  matching  the 
request  query. 

Accessing  the  get  /results  service  returns  a  Result  JSON  object.  For  example: 

REQ:  GET  /results/54de7a979565b5eecedl38ba 

RES:  { 

"model" :  . . . , 


"version" :  . . . , 
"settings" :  {...}, 
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"params" :  {...}, 

"outputs" :  {...}, 

"f inalOutputs" :  {...} 

I  ' 

To  reduce  the  amount  of  data  returned  for  a  query,  filtering  options  are  available  for  some 
services.  The  get  /results  and  get  /data  services  can  be  truncated  to  only  return  portions  of  a 
complete  Result  object.  For  example,  to  only  view  the  settings  for  a  particular  result: 

REQ:  GET  /results/54de7a97 9565b5eecedl38ba/settings 

RES:  {  "init" :  .  ..,  "max":  .  ..,  "step":  .  ..,  "method":  ...  } 

Similarly,  to  view  the  final  value  of  the  NRE  Cost  variable  for  all  results: 

REQ:  GET  /data/f inalOutputs/nreCost 

RES:  [ 

{"_id" : "54de7a979565b5eecedl38ba",  "nreCost":  ...}, 

{ "_id" : "54de7aad9565b5eecedl38bb" ,  "nreCost":  ...} 

I  ' 

The  get  /data  service  allows  filtering  using  URL  encoding  with  conditional  operators  in  the  table 
below.  For  example,  to  only  view  results  for  models  with  the  Change  Flag  and  META  Flag 
parameters  set  to  false  (0): 

REQ:  GET  /data/f inalOutputs/nreCost?params . changeFlag=0&params .metaFlag=0 

RES:  [ 

{"_id" : "54de7a979565b5eecedl38ba",  "nreCost":  ...}, 

{ "_id" : "54de7aad9565b5eecedl38bb" ,  "nreCost":  ...} 

I  ' 

.  Data  service  conditional  operators. 

Encoded  Operator  Operator  Condition 
=  =  Equal  to 

=$gte  >=  Greater  than  or  equal 

to 

=$gt  >  Greater  than 

=$lte  <=  Less  than  or  equal  to 
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=$lt 


< 


Less  than 


=$ne  !=  Not  equal  to 


To  submit  the  results  of  a  local  model  execution,  the  post  /results  service  expects  a  complete 
Result  object  (tag  optional)  as  the  request  body: 

|  REQ:  POST  /results 

{ 


"model" :  . .  .  , 

"version" :  . . . , 

"params" :  {  ...  } , 

"settings" :  {  ...  } , 

"outputs" :  {  ...  } , 

"finalOutputs" :  {  ...  } 

} 

RES:  {"_id":  54de7aae9565b5eecedl38c9 } 

Similarly,  the  post  /execute  service  requires  a  partial  Result  object  as  the  request  body: 

REQ:  POST  /execute 

{ 

"model" :  .  .  . , 

"version"  :  .  .  . , 

"params" :  {  ...  } , 

"settings" :  {  ...  } 

} 

RES:  {"_id":  54de7aae9565b5eecedl38ca} 

Backend  Implementation 

The  ISRM  services  are  implemented  in  a  Node.js169  runtime  environment  with  a  MongoDB170 
document-based  database  service.  These  technologies  define  JavaScript  as  a  common  language 
for  all  application  layers  including  the  client  (browser),  server,  and  database  document.  Although 
built  on  a  common  language,  browsers  do  not  yet  support  the  module  management  system  used 


169  Node.js  version  0.10.32.  http://nodejs.org,  accessed  24-Oct.  2014. 

170  MongoDB  version  2.6.5.  http://mongodb.org,  accessed  24-Oct.  2014. 
RequireJS  version  2.1.16.  http://requirejs.org,  accessed  18-Feb.  2015. 
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in  Node  so  MAS  and  ISRM  modules  implement  the  RequireJS171  interface  for  browser  and  server 
interoperability. 

The  Node  execution  uses  the  Express172Error!  Reference  source  not  found,  framework  to  define  a  simple 
web  server.  The  server  provides  routes  for  the  three  services  (request,  data,  and  execute  )  and 
otherwise  serves  static  content  as  HTML  pages  with  JavaScript  applications  (e.g.  standalone  tool, 
execution  service,  visualizations,  and  benchmark  tools).  The  server  application  accesses  a 
MongoDB  service  with  the  Mongoskin173  package.  Result  objects  are  stored  directly  in  MongoDB 
as  documents. 

The  figure  below  shows  results  of  a  performance  benchmark  comparing  local  (in  the  browser,  i.e. 
post  /result)  and  remote  (in  Node.js,  i.e.  post  /execute)  model  execution  services  using  JQuery 
AJAX  calls.  All  cases  log  time-varying  data  and  run  on  the  same  physical  machine  as  in  the 
standalone  benchmark.  Execution  services  requires  more  time  than  the  standalone  case  due  to 
database  insert/update  activities.  Local  model  execution  is  slightly  faster  for  META  projects  while 
remote  is  slightly  faster  for  non-META  projects  due  to  differences  in  data  transfer  quantity  arising 
from  project  durations.  In  other  words,  remote  model  execution  is  preferred  when  generating 
large  datasets  to  avoid  transmitting  data  over  services. 

The  service-based  methods  require  more  time  than  the  corresponding  standalone  methods  due 
to  additional  delays  for  client-web  server  and  web  server-database  server  communication  and 
database  actions.  These  examples  demonstrate  service  overhead  in  a  best  case  scenario  with  no 
network  latency  as  about  100  milliseconds  per  execution.  Overall,  the  service-based  application 
still  provides  a  rapid  execution  response,  allowing  about  five  executions  per  second.  Querying 
existing  data  is  much  faster  and  also  allows  caching  of  commonly-accessed  data  to  dramatically 
improve  performance. 


ISRM  execution  service  performance 
benchmark  under  four  conditions.  Error  bars 
show  95%  confidence  interval  over  100 
trials. 


171  RequireJS  version  2.1.16.  http://requirejs.org,  accessed  18-Feb.  2015. 

172  Express  version  4.11.2.  http://expressjs.com,  accessed  19-Feb.  2015. 

173  Mongoskin  version  1.4.12.  https://www.npmis.com/package/mongoskin,  accessed  19-Feb.  2015. 
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Browser-based  User  Interface 


The  ISRM  service-based  application  provides  Ul  modules  with  four  core  capabilities:  batch 
execution,  time  series  comparison,  tradespace  exploration,  and  sensitivity  analysis.  Each 
capability  is  embodied  as  a  separate  tool  accessing  services  with  JQuery  AJAX  methods. 

The  batch  execution  tool  in  the  figure  below  (left)  provides  full-factorial  design  experiment 
generation  with  local  or  remote  model  execution.  Users  select  and  vary  parameters  of  interest 
as  value  ranges.  The  tool  generates  list  of  runs  which  are  processed  with  post  /results  or  post 
/execute  services.  Additional  Ul  components  assign  optional  user  tags  to  particular  parameter  sets 
to  identify  models  of  interest. 

The  time  series  comparison  tool  in  the  figure  below  (right)  uses  the  get  /results  service  to  display 
the  simulation  log  data  of  a  selected  variable  under  various  conditions.  Individual  results  can  be 
assigned  color  schemes  which  persist  across  all  other  visualizations.  The  example  shown 
compares  the  time  history  for  the  NRE  Cost  stock  under  Optimistic  (blue).  Realistic  (red),  and 
META  (red)  conditions.  Users  can  select  particular  model  executions  to  display  and  change  the  y- 
axis  to  any  stock  or  flow  variable. 


Batch  execution  tool.  Users  can  run  full-factorial 
design  of  experiments  with  local  or  remote  model 
execution. 


ISRM  Time  Series  Visualization 


Time  series  visualization  tool.  Users  visualize  and 
compare  time  series  of  model  outputs. 


The  sensitivity  analysis  tool  in  the  figure  below  (left)  also  uses  the  get  /results  service  to  compare 
final  stock  values  or  parameters  as  percentage  differences  from  a  baseline  result.  The  example 
shown  compares  NRE  Cost  and  Project  Duration  for  Realistic  (red)  and  META  (green)  conditions 
to  the  baseline  Optimistic  (blue)  case.  Users  can  select  baseline  or  comparison  model  executions 
or  add  other  stock  or  parameter  variables  to  append  to  the  sensitivity  chart. 
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Finally,  tradespace  exploration  in  the  figure  below  (right)  uses  the  get  /data  service  to  visualize 
the  full  set  of  available  results  plotted  on  two  dimensions.  The  example  shown  visualizes  the 
Project  Duration-NRE  Cost  space  and  colors  non-META  results  red  and  META-enabled  results 
green.  Users  can  select  x-  and  y-axis  variables  from  available  stock  and  flow  variables.  A  filtering 
option  customizes  a  subset  of  results  to  display. 


ISRM  Sensitivity  Visualization 


!*C6«XWC4»*S4«*<  st 
r  it*  *****  «c«  a* 


Sensitivity  analysis  visualization  tool.  Users  visualize 
and  compare  final  model  outputs  to  a  baseline  case. 


Tradespace  exploration  tool.  Users  visualize  final 
model  outputs  on  a  two-dimensional  space. 


Service-based  Application  Limitations 

The  service-based  ISRM  application  only  considers  the  DFM  port  developed  under  Phase  1  and 
does  not  consider  changes  beyond  variable  parameter  values.  Node  provides  shell  and  file  system 
access  which  could  be  used  to  execute  external  models  in  future  work.  For  example,  a  properly- 
licensed  Vensim  application  with  command  line  access  may  be  coupled  with  the  Node  server  to 
provide  remote  model  execution.  This  approach  benefits  from  optimized  model  execution  in 
specialized  tools  but  would  suffer  from  time  delays  of  shell  and  file  system  access  rather  than 
manipulating  data  in  memory. 

This  application  also  only  considers  a  single,  non-malicious,  local  user  and  does  not  address  co¬ 
modification  of  data.  All  tools  query  and  interact  with  the  same  database  but  are  not  notified  of 
database  changes.  Extensions  to  multi-user  systems,  for  example  distributed  model  execution, 
may  require  additional  architectural  components  including  improved  server  security  and  API 
access  keys. 

MongoDB  allows  a  maximum  document  size  of  16  megabytes  which  limits  its  generalizability  to 
other  simulations  generating  large  datasets.  This  ISRM  application  stores  log  values  of  every  stock 
and  flow  variable  at  each  time  step  with  document  sizes  on  the  order  of  250  kilobytes  which  is 
considered  large  for  most  MongoDB  applications.  Models  with  larger  outputs  sets  may  be 
required  to  distribute  a  complete  results  document  across  several  constituent  documents. 


114 


While  required  for  client-server  interoperability,  RequireJS  reduces  client-side  performance  by 
preventing  the  browser  from  pre-fetching  JavaScript  files.  This  causes  a  delay  in  Ul  display  when 
a  page  is  first  loaded.  Although  not  yet  explored,  the  RequireJS  optimizer  or  other  methods  such 
as  Browserify174  may  provide  improved  performance  at  the  cost  of  compiling  JavaScript  files 
before  deploying  applications. 


Conclusion 

Intense  human-model  interaction  through  new  design  methods  and  tools  aim  to  improve 
perception  and  reduce  cost  and  schedule  to  realize  descriptively-complex  systems.  Applied  to 
system  project  management,  models  may  help  assess  alternative  system  development  processes 
and  resource  allocations.  ISRM  extends  past  work  to  develop  an  extensible  and  interactive 
approach  to  rapidly  analyze  sensitivity  to  various  factors  using  modern  web-based  technologies. 
Its  main  contributions  include  the  standalone  and  service-based  tools  as  prototypes  of  future 
interactive  model  development. 

The  standalone  ISRM  tool  implements  a  JavaScript  system  dynamics  (SD)  model  to  demonstrate 
model  execution  and  interaction  capabilities  in  a  browser-based  environment.  Performance 
benchmarks  show  model  executions  require  about  100  milliseconds  on  consumer  hardware.  The 
tool  provides  user  interface  components  similar  to  commercial  modeling  tools  using  open-source 
user  interface  component  libraries. 

The  service-based  ISRM  application  demonstrates  data  storage  and  query  services  using  the 
Node.js  platform  and  MongoDB  document-based  database.  Performance  benchmarks  show 
services  increase  execution  time  to  about  200  milliseconds  but  provide  rapid  access  to  stored 
and  cached  data  through  queries.  Demonstrative  applications  using  services  include  a  batch 
execution  tool,  time  series  visualization,  sensitivity  analysis,  and  tradespace  exploration. 

Mirroring  a  core  principle  of  other  web-based  technologies,  ISRM  tools  are  made  available 
through  online  repositories.  MAS  can  be  accessed  as  the  mas  Node  Package  Manager  (NPM) 
module  with  source  code  available  on  GitHub175.  The  standalone  and  service-based  ISRM  tools 
are  also  available  on  GitHub176.  Addendum  BThese  prototypes  could  be  used  to  help  evaluate 
future  method  and  tool  development  and  may  be  extended  in  potential  future  projects. 

There  are  two  broad  directions  for  potential  future  work.  One  direction  aims  to  improve  insights 
to  system  project  management  by  refining  the  underlying  design  flow  model.  In  particular,  any 
future  work  should  revisit  original  assumptions  such  as  initial  requirements  profile  and  workforce 
constraints  to  improve  generalizability  of  model  results.  This  activity  would  be  supported  by  a 


174  Browserify  version  8.0.3.  http://browserify.orfi,  accessed  20-Feb.  2015. 

175  MAS  version  0.0.3.  https://github.com/ptgrogan/mas,  accessed  20-Feb.  2015. 

176  ISRM  version  0.0.1.  https://github.com/ptgrogan/isrm,  accessed  20-Feb.  2015. 
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browser-based  SD  model  editor  to  replicate  a  larger  portion  of  the  functionality  of  proprietary 
tools  for  model-building  activities. 

Another  direction  of  future  work  could  be  to  mature  methods  developed  in  this  project.  The  MAS 
library  would  benefit  from  more  use  cases  to  implement  additional  functions  within  the  SD 
formalism  or  branch  into  other  formalisms  such  as  discrete  event  simulation  or  agent-based 
simulation.  Extensions  of  service-based  tools  may  address  other  limitations  previously  identified 
such  as  securing  and  synchronizing  data  across  multiple  concurrent  users.  Improvements  to  the 
prototype  applications  are  also  possible  to  improve  usability  and  efficiency  for  particular  tasks 
including  options  to  export  analysis  data  or  figures. 
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Addendum  A.  Design  Flow  Model  Documentation 


This  section  documents  components  the  Vensim-based  Design  Flow  Model  (DFM)  adapted  to  the  ISRM. 


Parameters 


ISRM  ID 

DFM  Name 

Description 

Units 

Default  Value 

schPressure 

Schedule  Pressure 

Multiplier  for  speed  of  several 
processes. 

1.5 

cogBandwidth 

Cognitive  Bandwidth 

Constraint  on  the  project  team's 
ability  to  consider  multiple  concepts 
at  the  same  time. 

requirements 

5 

metaFlag 

META  Flag 

Boolean  to  set  META  processes  on  (1) 
or  off  (0). 

1  (META) 

0  (non-META) 

changeFlag 

Change  Flag 

Boolean  to  set  change  generation  on 
(1)  or  off  (0). 

1 

designSpeed 

Design  Speed 

Multiplier  to  set  the  rate  of  design 
activities. 

0.25 

modlntegrity 

Component  Model 

Library  Integrity 

Fraction  of  the  component  model 
library  free  of  errors. 

0.8 

reStaffRatio 

RE  Staff  Ratio 

Productivity  of  requirements 

engineering  staff. 

requirements  per 

person-month 

10 

vvStaff Ratio 

VV  Staff  Ratio 

Productivity  of  verification  and 
validation  staff. 

tests  per  person- 
month 

20 

cgStaffRatio 

CG  Staff  Ratio 

Productivity  of  change  generation 
staff. 

changes  per  person- 
month 

4 

ceStaf f Ratio 

CE  Staff  Ratio 

Productivity  of  the  concept 
exploration  staff. 

concepts  per  person- 
month 

1000  (META) 

10  (non-META) 

aveLaborRate 

Average  Labor  Rate 

Mean  labor  cost  across  the  project 
team. 

$  per  person-month 

2000 

modCoverage 

Component  Model 

Library  Coverage 

Fraction  of  component  models 
already  existing  in  the  library. 

0.5  (META) 

0  (non-META) 

f racChangeReq 

Fraction  of  Changes  that 
affect  Requirements 

Fraction  of  changes  which  generate 
new  requirements. 

0.2 

f racProblemsCaught 

Fraction  of  Problems 
Caught  Initially 

Fraction  of  changes  caught  during 
design,  verification,  and  validation  to 
avoid  generation  of  changes. 

0.7 

archThroughput 

Architecture 

Enumeration 

Throughput 

Rate  of  potential  architecture 
generation. 

architectures  per 

month 

50  (META) 

10  (non-meta) 

Stocks 


ISRM  ID 

DFM  Name 

Description 

Units 

Initial 

Value 

Derivative  Formula 

reqDef ined 

Requirements 

Defined 

Number  of  requirements 
defined. 

requirements 

0 

reqElicit 

archExplored 

Architectures 

Explored 

Number  of  architectures 
explored. 

architectures 

0 

int (  conExploration) 

archRetained 

Architectures 

Retained 

Number  of  architectures 
retained. 

architectures 

0 

int  (  conExploration  - 

archFiltering) 

systemSpecs 

System 

Specifications 

Number  of  system 

specifications  generated. 

specifications 

0 

designlntegration 

testsPer formed 

Tests 

Performed 

Number  of  specifications 
tested. 

tests 

0 

verification 

pendChanges 

Pending 

Changes 

Number  of  changes 

pending  completion. 

changes 

0 

changeGen  -  changelmpl 

cumChanges 

Cumulative 

Changes 

Number  of  changes 

generated. 

changes 

0 

changeGen 
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reqValidated 

Requirements 

Validated 

Number  of  requirements 
validated. 

requirements 

0 

validation 

nreCost 

NRE  Cost 

Non-recurring  engineering 
cost. 

$ 

0 

spendRate 

projDuration 

n/a 

Duration  until  certificate  of 
completion  is  achieved. 

months 

0 

1  -  certCompletion 
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Flows 


ISRM  ID 

DFM  Name 

Description 

Units 

Value  Formula 

initReq 

Initial 

Requirements 

Initial  rate  of  requirements 
generation  from  design 
activities  shaped  by  schedule 
pressure  and  project  time. 

requirements 
per  month 

max(0,  -pow (  2  *  schPressure  * 

time  -  10  /  schPressure,  2)  + 

schPressure)  *  300) 

changeReq 

Change 

Requirements 

Rate  of  requirements 

generation  due  to  change 
implementation. 

requirements 
per  month 

f racChangeReq  *  changelmpl 

reqElicit 

Requirements 

Elicitation 

Rate  of  requirements 

generation  including  initial 
and  change  components. 

requirements 
per  month 

initReq  +  changeReq 

levAbs tract ion 

Level  of 

Abstraction 

Level  of  abstraction  for 
design  activities.  Non-META 
operates  at  a  single  level  of 
abstraction.  META  allows 
preliminary  work  at  other 
levels  of  abstraction 

determined  by  the  number  of 
requirements  and  cognitive 
bandwidth  of  the  team. 

if (  metaFlag  ==  1,  int (  log( 

reqDefined  +  1)  /  log ( 

cogBandwidth) )  ,  1) 

conSwitch 

Concept  Switch 

Boolean  to  turn  concept 
exploration  on  (1)  or  off  (0). 
Concept  exploration  is  always 
allowed  at  higher  levels  of 
abstraction  and  is  also 

allowed  under  low  levels  of 
requirements  elicitation. 

if  (  levAbstraction  >  1  |  | 

reqElicit  <  10,  1,  0) 

conExploration 

Concept 

Exploration 

Realized  exploration  rate  of 
potential  architectures. 

Exploration  is  limited  by  the 
maximum  exploration  rate 
and  the  architecture 

enumeration  throughput. 

Schedule  pressure  acts  as  a 
multiplier  for  concept 

exploration. 

architectures 
per  month 

schPressure  *  conSwitch  *  min ( 
explorationRate,  archThroughput) 

explorationRate 

Exploration  Rate 

Maximum  exploration  rate  of 
potential  architectures. 

META  explores  at  a  rate 
geometrically  proportional  to 
the  level  of  abstraction  and 
inversely  proportional  to  the 
time  and  number  of 

architectures  retained.  Non- 
META  explores  at  a  fixed  rate 
and  stops  when  one 
architecture  is  retained. 

architectures 
per  month 

if  (  metaFlag  ==  1,  (10  *  exp  ( 

levAbstraction))  /  ((0.1  *  time  + 

1)  *  (archRetained  +  1)),  if ( 

archRetained  >  1,  0,  10)) 

designSwitch 

Design  Switch 

Boolean  to  turn  design  and 
integration  on  (1)  or  off  (0). 
Design  is  allowed  at  higher 
levels  of  abstraction  or  if 
concept  exploration  ends 
with  at  least  one  retained 

architecture. 

if  (  levAbstraction  >  1  |  | 
conExploration  ==  0  && 
archRetained  >1,  1,  0) 

strComplexity 

Structural 

Complexity 

Measure  of  structural 
complexity  for  the  current 
design.  Complexity  is 

proportional  to  number  of 
requirements  and  inverse-log 
of  the  number  of 

architectures  explored 

reqDefined  /  (log(  sqrt( 

archExplored)  +  10)) 
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(elegance  through 

exploration). 

productivity 

Productivity 

Multiplier  for  the 

productivity  of  workers. 
Decreases  with  increasing 
novelty,  proportional  to 
requirements  defined,  and 
inversely  proportional  to 
complexity. 

(1  -  novelty  /  4)  /  2  +  min ( 

reqDefined  /  ( strComplexity  +  1), 

1) 

design Integration 

Design  and 

Integration 

Rate  of  design  and 

integration  specification. 

Proportional  to  schedule 
pressure.  Specifications 

come  from  requirements 
(proportional  to  design 
speed,  productivity,  and 
fraction  of  unspecified 
requirements  and  inversely 
proportional  to  novelty)  or 
change  implementation. 

specifications 
per  month 

schPressure  *  (designSpeed  * 

productivity  *  designSwitch  * 
sqrt (  (systemSpecs  +1)  *  max(0, 
reqDefined  -  systemSpecs))  /  (1  - 
modCoverage)  +  (1 

f racChangeReq)  *  changelmpl) 

novelty 

Novelty 

Fraction  of  new  components. 

1  -  modCoverage 

archFiltering 

Architecture 

Filtering 

Rate  to  filter  out  unwanted 

architectures. 

architectures 
per  month 

if (  archRetained  <=  1,  0,  delayl ( 
conExploration,  1) ) 

testSwitch 

Test  Switch 

Boolean  to  turn  testing  on  (1) 
or  off  (0). 

if  (  levAbstraction  >  1  |  | 

designlntegration  <  10),  1,  0) 

verification 

Verification 

Rate  of  specification  tests. 
Proportional  to  schedule 
pressure  and  productivity. 

tests  per 

month 

if  (  testsPerf ormed  <  systemSpecs, 
schPressure  *  testSwitch  * 
productivity  *  sqrt( 
(testsPerf ormed  +  1)  * 
(systemSpecs  -  testsPerf ormed) ) , 

0) 

changeGen 

Change 

Generation 

Rate  of  change  generation. 

changes  per 
month 

changeFlag  *  max (  (reqDefined  +  1 
-  reqValidated)  /  (reqDefined  + 
1),  0)  *  delayl (  strComplexity  / 
(reqDefined  +  1)  *  (  (1  - 
fracProblemsCaught)  * 
(verification  +  validation  + 
designlntegration)  +  delayl ( 
metaFlag  *  novelty  *  (1  - 
modlntegrity)  * 
designlntegration,  4)),  1)  *  if( 
metaFlag  ==  1,  novelty  +0.5,  1) 

changelmpl 

Change 

Implementation 

Rate  of  changes 

implemented. 

changes  per 
month 

smoothl (  delayl (  changeGen,  0.5, 

1) 

validationSwitch 

Validation 

Switch 

Boolean  to  turn  validation  on 
(1)  or  off  (0). 

if (  levAbstraction  >  1  | | 

verification  <  10),  1,  0) 

validation 

Validation 

Rate  of  requirements 

validation.  Proportional  to 
schedule  pressure  and 

productivity. 

requirements 
per  month 

if (  reqValidated 

testsPerformed,  schPressure  * 

validationSwitch  *  productivity  * 
sqrt ( (  reqValidated  +  1)  * 

(testsPerformed  -  reqValidated) ) , 

0) 

certCompletion 

Certificate  of 

Completion 

Number  of  certificates  of 
completion  issued. 

if (reqValidated  >  0.999  * 

reqDefined  &&  pendChanges  <  1,  1, 

0) 

diStaf f Ratio 

Dl  Staff  Ratio 

Productivity  of  design  and 
integration  staff. 

parts  per 

person- 

month 

4  /  novelty 

sysEngineers 

System 

Engineers 

Number  of  system  engineers. 

people 

int  (  conExploration  / 
ceStaffRatio  +  reqElicit  / 
reStaf fRatio) 

designers 

Designers 

Number  of  designers. 

people 

int (  designlntegration  / 
diStaffRatio  +  changeGen  / 
cgStaf fRatio) 

testers 

Testers 

Number  of  testers. 

people 

int (  (validation  +  verification) 

/  vvStaf fRatio) 

spendRate 

Spending  Rate 

Rate  of  spending  money. 

$  per  month 

(sysEngineers  +  designers  + 
testers)  *  aveLaborRate 
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Addendum  B.  Installation  and  Configuration 


Portions  of  this  guide  are  based  on  the  following  tutorials  from  Christopher  Buecheler: 

•  http://cwbuecheler.com/web/tutorials/2013/node-express-mongo/ 

•  http://cwbuecheler.com/web/tutorials/2014/restful-web-app-node-express-mongodb/ 


Software  Repository 

A  GitHub  software  repository  holds  the  source  code  for  both  the  standalone  and  service-based  ISRM 
tools.  To  connect,  follow  the  following  instructions. 

1.  Install  a  Git  client:  https://help.github.com/articles/set-up-git 

2.  Clone  the  ISRM  repository:  https://github.com/ptgrogan/isrm 


Server  Configuration 

The  service-based  ISRM  tool  requires  a  properly-configured  server.  This  section  describes  the  process  to 
set  up  required  software  on  Windows.  Other  operating  systems  have  not  been  tested  but  should  work. 

1.  Download  and  install  the  latest  32-bit  or  64-bit  version  of  Node.js  (version  0.10.35  as  of  writing): 

http://nodejs.org/download/ 

•  Node.js  is  released  under  the  MIT  license. 

2.  Download  and  install  the  latest  64-bit  (preferred)  version  of  MongoDB  (version  2.6.6  as  of 
writing):  http://www.mongodb.org/downloads 

•  MongoDB  is  released  under  the  Free  Software  Foundation  GNU  AGPL  v3.0  license. 

•  Note:  the  64-bit  version  is  preferred  to  allow  databases  with  more  than  2  GB  of  data. 

3.  Start  the  database  service.  Open  a  command  console  and  navigate  to  the  MongoDB  install 
directory  and  execute  the  following  command  to  start  the  database  service. 

cd  C:\path\to\MongoDB\bin 

mongod  --dbpath  C:\path\to\isrm\data 

This  command  starts  the  MongoDB  service  on  the  default  port  27017  using  the  directory  path 

C:\path\to\isrm\data  to  store  documents. 

4.  Install  the  Node.js  service.  Open  another  command  console  and  navigate  to  the  ISRM  Server 
directory.  Execute  the  following  NPM  command  to  install  dependencies. 

cd  C:\path\to\isrm 
npm  install 

This  command  installs  packages  listed  in  the  file  package,  j son.  If  you  receive  an  ENOENT  error 
message,  manually  create  the  directory  at  the  corresponding  location  (likely 

C:  \Users\username\AppData\Roaming\npm)  and  re-issue  install  command. 
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5.  Start  the  Node.js  service: 

npm  start 

This  command  starts  the  Node.js  service  on  default  port  3000. 


Connect  to  http://localhost:3000  in  any  web  browser.  The  default  page  in  Figure  50  shows  a 
dashboard  of  available  tools  including  benchmarking  applications. 


ISRM:  Interactive  Schedule  Reduction  Model  -  Google  Chrome 


Q  localhost:3000/index.html 


ISRM  Dashboard 


STANDALONE  TOOL 


EXECUTION  SERVICE 


Demonstrates  the  standalone  variant  of  the  ISRM  tool.  This  application  runs 
completely  within  a  browser  and  supports  local  model  execution  for  fixed 
scenarios.  The  benchmark  application  quantitatively  evaluates  execution 
performance  for  scenarios  with  and  without  META  design  processes  and 
simulators  with  and  without  data  logging.  The  standalone  application  provides  a 
graphical  user  interface  illustrating  model  structure  and  documentation,  time  history 
output  plots,  and  export  capabilities  (CSV,  JSON  formats). 

Demonstrates  a  service-based  variant  of  the  ISRM  tool.  This  application  runs  as  a 
client  service  in  a  browser  and  a  separate  data  service  provides  database  access 
to  stored  results  and  remote  execution.  The  benchmark  application  quantitatively 
evaluates  local  and  remote  execution  performance  for  scenarios  with  and  without 
META  design  processes.  The  execution  service  application  supports  full-factorial 
design  studies  varying  key  model  parameters  with  local  and  remote  model 
execution.  Three  visualizations  support  graphical  tradespace  exploration, 
sensitivity  analysis,  and  time  series  comparison. 


Disclaimer:  This  project  is  based  upon  work  supported,  in  whole  or  in  part,  by  the  U.S.  Department  of  Defense  through  the 
Systems  Engineering  Research  Center  (SERC)  under  Contract  HQ0034-13-D-0004.  SERC  is  a  federally  funded  University 
Affiliated  Research  Center  managed  by  Stevens  Institute  of  Technology.  Any  opinions,  findings  and  conclusions  or 
recommendations  expressed  in  this  material  are  those  of  the  author(s)  and  do  not  necessarily  reflect  the  views  of  the  United 
States  Department  of  Defense. 


Copyright©  2015  Paul  T.  Grogan,  Massachusetts  Institute  of  Technology.  All  Rights  Reserved 


Figure  50  Default  ISRM  dashboard. 
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Addendum  C.  Source  Code  Guide 


This  section  provides  an  overview  of  the  source  code  for  the  standalone  and  service-based  tools.  The 
ISRM  repository  follow  this  general  structure: 


isrm\ 

o 

bin\ 

Node  scripts  to  start  the  application 

o 

(dataX ) 

MongoDB  database  files  (create  after  install) 

o 

(node  modulesX) 

Third-party  Node  modules  (created  on  install) 

o 

routesX 

Express  service  routes 

o 

www\ 

Static  web  files 

o 

app. js 

Express  web  server  application 

o 

package . j  son 

Node  package  definition 

o 

README . md 

Information  markdown  file 

Standalone  Tool 

The  standalone  ISRM  tool  emphasizes  the  following  files: 


isrm\www\ 

O  images \ 

O  scriptsX 

■  app\ 

•  standalone . j s 

■  lib\ 

■  common . j  s 

■  standalone . j s 
O  stylesX 

O  standalone.html 


Images  for  the  user  interface 
Directory  for  JavaScript  scripts  and  libraries 
Directory  for  JavaScript  application  scripts 
Standalone  application  script 
Directory  for  JavaScript  libraries 
Common  configuration  for  RequireJS 
RequireJS  loader  for  standalone  tool 
Directory  for  CSS  styles 
Standalone  tool  HTML  page 


Open  the  file  standalone.html  in  any  web  browser  to  directly  access  the  standalone  tool.  This 
application  and  the  related  benchmark-standaione.html  work  without  the  Node  web  server  or 
MongoDB  database  components  as  they  do  not  access  the  ISRM  services. 


Service-based  Application 

The  service-based  ISRM  tool  emphasizes  the  following  files: 


isrm\ 

O  bin\ 

O  data\ 

O  node_modules\ 

O  routesX 

■  data.js 

■  execute. js 

■  results. js 


Node  scripts  to  start  the  application 
MongoDB  database  files  (create  after  install) 
Third-party  Node  modules  (created  during  npm  install) 
Express  Service  routes 
Data  services 
Execute  services 
Results  services 
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o 


o 

o 


www\ 


images \ 
scriptsX 
•  app\ 


Static  web  files 

Images  for  the  user  interface 

Directory  for  JavaScript  scripts  and  libraries 

Directory  for  JavaScript  application  scripts 


o 

execution . j  s 

Batch  execution  tool 

o 

visualize-history.js 

Time  series  visualization  tool 

o 

visualize- sensitivity. j  s 

Sensitivity  analysis  tool 

o 

visualize-tradespace . j  s 

Tradespace  exploration  tool 

lib\ 

common . j  s 
execution . j  s 
visualize- hi story. j  s 
visualize- sensitivity . j  s 
visualize- trade space . j  s 
styles\ 

execution . html 
index . html 

visual ize-hi story. html 
visualize-sensitivity.html 
visualize-tradespace . html 


Directory  for  JavaScript  libraries 
Common  configuration  for  RequireJS 
RequireJS  loader  for  batch  execution 
RequireJS  loader  for  time  series  tool 
RequireJS  loader  for  sensitivity  tool 
RequireJS  loader  for  tradespace  tool 
Directory  for  CSS  styles 
Batch  execution  HTML  page 
ISRM  dashboard  page 
Time  series  visualization  HTML  page 
Sensitivity  analysis  HTML  page 
Tradespace  exploration  HTML  page 


app . : s 

package . j  son 


Express  web  server  application 
Node  package  definition 


Once  the  Node  web  server  and  MongoDB  database  service  are  running,  access  the  dashboard  page 

(index,  html)  at  http://localhost:3000  in  any  web  browser 
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Moving  Forward  to  Phase  Three 


•  The  research  team  will  be  using  knowledge  and  information  gained  in  Phase  1  and  Phase  2  to 
focus  ongoing  efforts  in  Phase  3  to  further  explore  the  identified  IMCSE-related 
considerations  within  four  key  areas,  and  the  challenges  and  opportunities  at  their 
intersection. 

•  The  Pathfinder  Workshop  Report  (Appendix  A)  will  be  released  to  elicit  comments  and 
recommendations,  augmented  by  discussions  with  selected  subject  matter  experts.  This  will 
feed  into  creating  a  collaboratively-derived  research  agenda.  A  research  roadmap  will  be 
derived  in  collaboration  with  other  SERC  researchers  and  the  broader  systems  community.  A 
leadership  summit  may  be  conducted  to  support  validation  of  research  priorities, 
recommend  pathways  to  accelerate  research  progress,  and  enable  transition  to  the  systems 
community. 

•  The  team  will  perform  research  to  mature  the  approach  for  evaluating  systems  under 
dynamic  uncertainty,  with  further  development  of  the  extended  framework  to  for  interactive 
capability  and  scaling  to  big  data.  This  work  extends  the  Phase  2  effort  on  a  demonstration 
prototype,  the  Interactive  Value-Driven  Tradespace  Exploration  and  Analysis  Suite  (IVTea 
Suite)  that  applied  IMCSE  principles  to  enhance  the  user  interface,  data  handling  and  analysis 
widgets.  In  Phase  3  the  research  team  will  enhance  the  method  and  degree  of  interactive 
capability,  focusing  specifically  on  the  Epoch-Era  Analysis  (EEA)  method,  a  novel  method  for 
value-driven  tradespace  exploration  and  analysis.  The  maturing  prototype  framework  with 
associated  supporting  tools  will  be  applied  to  a  case  analysis  including  various  types  of 
uncertainties.  This  case  application  will  be  used  to  elicit  feedback  on  relevance,  ease  of  use, 
feasibility  and  tractability  of  data  scaling  and  visualization  techniques.  The  research  team  will 
extend  the  Phase  2  prototype  for  interactive  Epoch-Era  Analysis  and  test  it  using  a  case 
application,  along  with  preliminary  supporting  infrastructure,  which  will  then  be  used  to 
inform  the  transition  strategies,  additional  case  application  and  prototype  user  testing.  The 
team  will  build  on  the  Phase  2  work  on  value  model  trades  to  further  evolve  the  framework 
and  process,  and  apply  this  through  an  expanded  case  application. 

In  Phase  3,  the  research  team  will  build  on  prior  phase  results  to  further  evolve  the  framework 
and  process  for  conducting  value  model  choice  and  tradeoffs  and  apply  this  through  an 
expanded  case  application  set,  to  validate  the  framework  and  identify  workflow 
considerations.  In  this  phase,  the  model  choice  and  tradeoff  framework  will  be  expanded 
including  demonstration  cases  beyond  value  models  (to  include  trading  of  other  types  of 
models  including  performance  and  cost  models).  The  expanded  framework  will  consider 
alternative  use  cases  for  the  impact  of  model  choice  and  tradeoffs  on  decision-making.  For 
example,  this  includes  the  context  of  multi-stakeholder  negotiations  using  tradespace 
exploration,  where  the  data  source(s)  (i.e.  "models")  strongly  impact  the  trust  and  framing  of 
the  shared  decision  problem. 

•  The  research  team  will  continue  to  investigate  the  cognitive  and  perceptual  considerations  in 
human-model  interaction,  a  topic  for  which  little  research  exists,  though  there  is  a  body  of 
knowledge  to  draw  from.  Preliminary  heuristics/design  principles  will  be  gathered,  adapted 
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for  human-model  applicability,  and  synthesized  as  a  draft  guidance  document.  The  guide  will 
be  shared  for  review  and  comment  by  model  developers,  users  and  model-based  software 
designers,  toward  publication  of  a  validated  set  of  guiding  principles  for  effective  human- 
model  interaction  during  Phase  4.  A  goal  is  to  involve  one  or  more  SERC  collaborators  as 
transition  partners,  to  pilot  use  of  the  guiding  principles  during  Phase  3. 

•  The  research  team  will  use  the  results  of  Phase  1  and  Phase  2,  along  with  ongoing  Phase  3 
research  interim  results,  to  develop  several  publishable  papers  for  journal  and  conference 
submissions.  Evolving  prototype  MPTs  will  be  shared  and  demonstrated  at  one  or  more  SERC- 
related  events  during  Phase  3,  including  the  CSER  2015.  The  research  team  will  continue 
active  knowledge  exchanges  with  several  other  SERC  researchers  performing  related  work, 
where  IMCSE  outcomes  can  inform  and/or  be  applied  in  their  work. 
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Transition  Objectives 


An  imperative  for  SERC  research  teams  is  the  effective  transition  of  research  to  practice,  including 
transfer  of  new  knowledge,  research  findings,  and  new  MPTs  to  members  of  the  community  of 
interest.  In  Phase  1,  we  have  developed  our  initial  plan  toward  this  objective,  and  in  Phase  2  have 
been  implementing  the  plan.  The  plan  includes  identifying  and  working  with  transfer  partners 
and  research  collaboration  partners.  Phase  3  will  continue  this  work. 

•  In  Phase  2  IMCSE  research  on  current  state  of  the  art  and  practice  was  shared  among 
participants  in  the  Pathfinder  Workshop  held  on  20  January  2015.  The  report  will  be 
disseminated  for  additional  feedback  and  there  will  be  subsequent  exchanges  extending 
from  this  workshop  via  teleconferences  and  meetings. 

•  The  Pathfinder  Project  Report  synthesizes  the  observations,  findings  and 
recommendations  in  current  art  and  practice,  research  needs,  emerging  research,  and 
an  envisioned  ideal  world.  This  report  issued  at  the  end  of  Phase  2,  will  undergo  a 
review/comment  period  in  Phase  3.  During  Phase  3  the  research  team  will  submit  a 
paper  on  the  emerging  research  agenda. 

•  During  Phase  3,  the  research  team  will  evolve  the  list  of  individuals  and  organizations  to 
be  contacted  for  inputs  for  the  activity  of  developing  a  collaboratively-derived  research 
agenda.  A  working  paper  initiated  during  Phase  2  to  capture  the  approach  and  lessons 
learned  in  creating  this  agenda  will  be  evolved  during  Phase  3.  A  journal  submission  will 
subsequently  be  prepared  from  the  working  paper  at  the  end  of  Phase  3. 

•  Results  of  the  Interactive  Schedule  Reduction  Model  and  Interactive  Epoch-Era  Analysis 
were  shared  with  the  broader  SERC  community,  in  selected  meetings  and  workshops, 
including  NDIA  SE  2014  and  INCOSE  IW15.  A  paper  on  each  of  these  projects  was 
accepted  to  CSER  2015,  and  will  be  presented. 

•  During  Phase  3,  the  ongoing  work  on  the  Interactive  Epoch-Era  Analysis  and  the  Model 
Choice  and  Tradeoffs  will  be  shared  and  demonstrated  at  one  or  more  SERC  events. 

•  One  or  more  selected  conference  publications  developed  during  Phase  2  will  be  evolved 
to  journal  submissions  during  Phase  3. 

•  Aspects  of  the  IMCSE  research  were  presented  at  the  2015  Complex  Systems  Design  & 
Management  (CSD&M)  Conference,  and  specific  discussions  were  held  on  related 
research  efforts  ongoing  in  Europe. 

•  Throughout  Phase  3,  synergies  with  other  SERC  tasks  were  identified  and  leveraged  to 
transition/implement  resulting  capabilities  of  this  project,  as  well  as  to  provide  relevant 
information  to  impact  the  work  of  other  researchers.  The  research  team  is  actively 
exchanging  information  with  other  SERC  researchers  with  research  relevant  to  IMCSE. 

•  The  research  team  discussed  the  work  with  leaders  of  the  INCOSE  Model-Based  Systems 
Engineering  initiative  during  the  INCOSE  IW  2015,  looking  for  points  of  connection  in  the 
work. 

•  In  Phase  2,  the  research  team  has  identified  potential  transition  partners  within  and 
external  to  SERC  for  user-testing  of  methods  and  prototypes.  During  Phase  3,  one  or 
more  of  these  partnerships  will  be  established. 
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Conclusion 


The  research  team  completed  the  Phase  1  effort,  taking  place  over  a  four  month  period  and  has 
now  completed  the  Phase  2  effort,  taking  place  over  a  five  month  period.  The  activities  within 
the  three  thrusts  -  foundations,  fundamentals,  and  applications  -  are  on-track  for  the  overall 
goals  specified  for  Phase  1  and  Phase  2,  and  readiness  to  transition  to  Phase  3  has  been  achieved. 

At  the  end  of  Phase  2  the  team  has  converged  on  key  research  topics  and  specific  activities  in 
support  of  the  broader  IMCSE  objectives.  The  pathfinder  workshop  has  validated  the  need  for 
research  on  IMCSE,  and  resulted  in  a  significant  set  of  observations,  findings  and 
recommendations.  This  has  provided  a  sound  foundation  for  further  knowledge  gathering  and 
moving  forward  with  efforts  to  build  a  community  around  the  IMCSE. 

Phase  2  has  demonstrated  several  ideas  and  technology  strategies  for  interactive  model-centric 
activities,  and  specific  plans  are  in  place  for  Phase  3  to  build  on  these  findings.  The  Interactive 
Epoch-Era  Analysis  activity  has  progressed  and  a  plan  is  in  place  for  the  Phase  3  work. 

Phase  2  investigations  have  revealed  the  importance  of  understanding  the  perceptual  and 
cognitive  considerations  for  human-model  interaction,  and  a  Phase  3  effort  has  been  formulated 
to  address  this  area.  The  research  team  will  work  toward  evolving  research  findings  to  practical 
guidance  in  the  form  of  heuristics  and  design  principles. 
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